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PREFACE 


This  document  is  the  final  technical  report  (CDRL  Item  A002)  for 
the  Impact  of  MPP  on  System  Development,  Contract  F30602-76-C-0095.  It 
presents  the  results  of  an  evaluation  of  the  impact  of  modern 
programming  practices  (MPP)  applied  to  TRW's  Ballistic  Missile  Defense 
(BMD)  Systems  Technology  Program  (STP)  software  development.  The  report 
provides  an  overview  of  STP  including  a reconstructed  chronology  of 
significant  events  and  a description  of  the  STP  software  development 
environment.  It  also  presents  detailed  results  of: 

• Identification,  selection  and  definition  of  appropriate 
STP  practices  to  be  evaluated 

• Investigation  of  the  impact  of  selected  MPP  on  software 
development  cost,  schedule  and  quality 

• Definition  and  preliminary  application  of  techniques 
and  tools  comprising  methodology  for  comparing  TRW 
practices  with  those  used  by  other  contractors  on  other 
projects. 

The  report  was  prepared  by  J.  R.  Brown  with  significant 
contributions  by  J.  W.  Dowdee,  D.  E.  Hacker,  G.  R.  Keludjian, 

D.  S.  Lee,  M.  Lipow,  G.  R.  Paxton  and  T.  R.  Savage.  Additional 
assistance  and  guidance  came  from  the  MPP  Project  Review  Committee 
which  included  B.  W.  Boehm,  J.  M.  Dreyfus,  F.  S.  Ingrassia, 

E.  C.  Nelson  and  F.  G.  Spadaro. 
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EVALUATION 


This  report  describes  the  software  development  technology  and 
management  practices  employed  on  a large  and  complex  system  development 
program  by  TRW. 

The  intent  of  the  RADC  program  to  which  this  document  relates, 

TPO  V/3.4,  is  to  describe  and  assess  software  production  and  management 
tools  and  methods  which  significantly  impact  the  timely  delivery  of 
reliable  software. 

The  study  contract  is  one  of  a series  of  six  with  different  firms 
having  the  similar  purpose  of  describing  a broad  range  of  technigues 
which  have  been  found  beneficial. 

RADC  is  engaged  in  promoting  utilization  of  Modern  Programming 
Technology,  also  called  Software  Engineering,  especially  in  large 
complex  Command  and  Control  software  development  efforts. 

ROGER  W.  WEBER 
Project  Engineer 


PAGE  1 


1 


1.0  INTRODUCTION  AND  SUMMARY 


1 . 1 Background 

For  a number  of  years  TRW  and  other  contractors  have  been  develop- 
ing ways  to  do  a better  job  of  software  development.  The  primary  impetus 
for  this  effort  has  come  from  increasing  demands  for  error-free  (or  at 
least  highly  reliable)  software,  from  the  need  for  software  which  is 
easily  (i.e.,  quickly  and  inexpensively)  modified  to  meet  newly  imposed 
requirements,  and  from  recognition  that  a cost-effective  software  engi- 
neering discipline  (not  the  "black  art"  of  old)  can  be  achieved  through 
identification,  application  and  evaluation  of  improved  production 
practices  [1 , 2,  3,  4]. 

It  is  generally  believed  that  the  quest  for  improved  Modern  Pro- 
gramming Practices  (MPP),  coupled  with  many  opportunities  to  apply, 
evaluate  and  further  improve  upon  them  has  brought  about  a significant 
advancement  of  modern  software  technology  [5,  6],  In  further  pursuit  of 
the  goal  of  a cost-effective  software  engineering  discipline,  the  Air 
Force  Rome  Air  Development  Center  (RADC)  recognized  the  need  for  an 
investigation  of  the  actual  met  its  of  a broad  range  of  established  and 
newly  proposed  practices. 

i In  the  last  few  years  some  trends  in  DoD  procurement  philosophy 

have  placed  strong  attention  on  the  evolution  and  application  of  MPP  by 

I software  contractors  [7,  8].  The  most  noticeable  trends  are:  1)  an 

unprecedented  dual  interest  in  both  the  cost  of  producing  software  and 
the  quality  of  the  end  product,  and  2)  requests  for  proposals  which 
require  bidders  to  identify  and  propose  application  of  specific  practices 
that  address  and  alleviate  the  problems  that  have  plagued  past  software 
development  activities. 

The  combined  committments  of  both  the  DcD  and  major  software  con- 
tractors to  accelerate  the  definition  and  application  of  improved  pro- 
duction practices  have  resulted  in  a dramatic  increase  in  the  use  of  MPP 
on  some  large  scale,  highly  critical  software  procurements.  A prominent 
example  is  the  TRW  development  of  the  Data  Processing  Subsystem  software 
for  the  Ballistic  Missile  Defense  Systems  Technology  Program  (BMD  STP). 
This  program  was  previously  named  and  is  frequently  referred  to  as  Site 
Defense  (SD).  The  importance  of  the  Data  Processing  Subsystem  to  ultimate 
successful  performance  of  the  system  has  placed  extraordinary  demands  on 
TRW  as  the  software  development  contractor.  To  ensure  the  production  of 
software  that  will  meet  the  system  requirements,  TRW  invented  a unique, 
rigorous,  and  highly  disciplined  software  development  process  [9].  To 
overcome  many  of  the  usual  problems  of  software  development  and  to  ensure 
the  development  of  error  resistant,  readily  understood  and  easily  main- 
tained software,  a comprehensive  set  of  standard  programming  practices  was 
established  and  firmly  imposed  [10].  To  ensure  adherence  to  the  disci- 
plined development  methodology  and  to  support  the  production  of  high 
quality  software  within  reasonable  schedule  and  cost,  software  developers 
were  supplied  with  a set  of  automated  tools  to  aid  design,  code,  debug, 
test,  and  documentation  activities  [11,  12,  13]. 
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In  addition,  TRW  monitored  the  "vital  signs"  of  project  activity  in 
assessing  the  effectiveness  of  the  development  process,  the  standards  and 
the  support  tools.  This  required  collection  of  data  on  the  software  develop- 
ment activity  and  frequent  reviews  of  project  performance.  Based  on  these 
reviews  and  on  analyses  cf  the  data,  changes  were  made  to  improve  the 
development  process,  the  programming  standards,  and  the  tools  used  to 
support  development.  This  resulted  in  the  evolutionary  development  of  a 
production  methodology  most  appropriate  for  the  STP  software  development 
activity.  With  the  encouragement  of  the  procuring  agency  (U.S.  ARMY/ 

BMDSCOM)  and  the  STP  prime  contractor  (McDonnell  Douglas  Astronautics 
Company),  TRW  actively  took  the  lead  in  developing  and  effectively  applying 
modern  programming  practices.  The  investment  in  improved  practices  has 
paid  off,  and  the  STP  software  development  is  on  schedule  and  within  cost, 
and  has  experienced  a significant  reduction  in  the  number  of  errors  found, 
per  line  of  code,  in  integration  testing  of  completed  increments  of  the 
deliverable  software. 
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Objective,  Scope  and  Problems  of  MPP  Stud> 


In  view  of  the  RADC  interest  in  improving  the  technology  applied  to 
future  Air  Force  software  development,  TRW  proposed  to  conduct  a study  of 
MPP,  concentrating  on  those  used  in  the  STP  software  development  activity. 
The  overall  goal  of  the  study  was  to  provide  an  evaluation  of  the  impact 
of  STP  programming  principles,  guidelines,  standards,  tools  and  management 
practices  to  determine  their  potential  for  application  to  Air  Force  soft- 
ware development.  RADC  contracted  with  TRW  to  study  the  Impact  of  MPP  on 
System  Development  (Contract  No.  F30602-76-C-0095) , and  this  report  con- 
stitutes the  final  contractual  deliverable  of  the  completed  study. 

The  primary  objectives  of  the  study  were  to: 

• Assess  the  impact  of  modern  programming  practices  (MPP) 
applied  to  the  TRW  STP  software  development  activity. 

« Develop  a methodology  for  comparison  of  STP  MPP  with 
alternative  MPP  implementations  used  on  other  projects 
by  other  contractors. 

In  order  to  fully  achieve  these  objectives  we  needed  to  obtain  valuable 
insights  into  the  relative  merits  of  individual  practices  and  to  devise 
and  demonstrate  a technique  to  evaluate  the  combined  effectiveness  of 
selected  practices  when  applied  to  projects  of  varying  type,  size,  and 
complexity. 
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In  the  MPP  Study  Statement  of  Work  it  was  hypothesized  that:  Soft- 
ware requirements  definition,  design,  implementation,  evaluation  and  docu- 
mentation rules,  if  rigorously  defined  and  applied,  and  supported  by 
modern  techniques  and  tools,  make  possible  the  production  of  higher  quality 
software  at  lower  than  usual  cost.  One  of  the  primary  objectives  of  the 
study  was  to  either  confirm  or  refute  this  general  hypothesis  specifically 
with  regard  to  the  extensive  application  of  modern  practices  to  the  TRW  STP 
software  development  activity.  That  is,  TRW  was  expected  to  obtain  and 
evaluate  sufficient  relevant  evidence  about  the  effectiveness  of  applied 
practices  to  permit  an  objective  conclusion  as  to  the  actual  validity  of 
the  hypothesis  (in  the  STP  case)  and  the  probable  validity  of  the 
hypothesis  in  general. 

In  attempting  to  carry  out  this  task,  TRW  was  faced  with  several 
formidable  problems.  First,  the  hypothesis,  as  stated  above,  is  extremely 
general  and  ambiguous.  The  state-of-the-art  of  software  engineering  has 
not  yet  produced  a useful  definition  of  "higher  quality  software",  and, 
although  much  has  been  done  recently  toward  development  of  software  cost 
estimation  methods,  the  emerging  cost  models  are  as  yet  imprecise,  awkward 
to  use,  and  limited  in  scope  of  application.  As  a result,  there  is  not 
yet  a commonly  understood  ana  accepted  meaning  of  "higher  quality  software" 
and  "usual  cost"  upon  which  a test  of  the  hypothesis  can  be  based. 

The  second  major  problem  has  to  do  with  the  availability  of  relevant 
data  to  be  used  in  testing  the  hypothesis.  Although  a great  deal  of  data 
on  the  STP  software  development  activity  was  collected  and  available  to 
support  the  MPP  study,  the  identification  of  truly  relevant  data  and  trans- 
lation into  credible  evidence  concerning  the  hypothesis  was  seen  to  be  a 
costly  and  probably  imprecise  and  inconclusive  task  unless  planned  and 
approached  in  a highly  selective  fashion. 

1 . 3 Overview  of  Study  Results 

The  TRW  study  of  the  Impact  of  MPP  on  System  Development  is  com- 
plete and,  with  the  completion  of  this  report,  the  requirements  of  the 
Statement  of  Work  are  satisfied.  In  striving  to  overcome  the  possible 
adverse  effects  of  the  above  mentioned  problems,  TRW  defined  and  imple- 
mented an  innovative  MPP  impact  assessment  approach  which  included: 

• Formulation  of  many,  more  definitive  sub-hypotheses  involving 
the  impact  of  individual  practices  on  characteristics  of  software 
and  the  software  development  process. 

• Accomplishment  of  a modified  Delphi  exercise  including  several 
surveys  of  STP  personnel  to  permit  both  relative  quantification 
and  testing  of  the  sub-hypotheses. 

• Highly  selective  examination  of  STP  records  and  documentation 
for  further  testing  of  certain  sub-hypotheses. 

t Evaluation  of  the  overall  hypothesis  through  combination  of 
sub-hypothesis  test  results. 
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Each  of  the  surveys  was  based  on  a list  of  practices  produced  by  refining 
and  extending  the  MPP  listed  in  the  TRW  MPP  Study  Proposal.  The  surveys 
were  designed  to  achieve  a balance  of: 

• desired  objectivity  in  formulation  and  testing  of  the 
hypotheses,  and 

• essential  reliance  on  actual  experience  and  use  of 
qualified  engineering  judgement  in  evaluating  MPP  impact. 

The  participants  in  the  impact  evaluation  study  activity  were  selected  from 
a cross-section  of  STP  management  and  performer  personnel  and  other  key  TRW 
personnel  including  representatives  from  staff  groups  responsible  for  soft- 
ware technology  planning  and  research. 

The  surveys  were  key  to  the  success  of  the  MPP  impact  assessment 
activity.  In  that  they  were  conducted  in  the  STP  context.  Section  2 of 
this  report  provides  a detailed  discussion  of  STP  software,  computing 
facilities,  personnel  and  a chronology  of  significant  project  events.  Sec- 
tion 3 presents  the  study  approach  in  terms  of  detailed  task  activities, 
discusses  the  steps  taken  in  selecting  a final  set  of  MPP  for  study,  and 
describes  in  considerable  detail  the  approach,  analyses  and  results  of  the 
surveys.  Section  4 follows  with  a discussion  of  the  need  for  a methodo- 
. logy  to  permit  comparative  evaluation  of  alternative  MPP  implementations. 

Section  4 further  discusses  the  feasibility  and  difficulty  of  arriving  at 
l an  acceptable  and  widely  useful  comparison  methodology,  and  presents  a 

proposed  comparison  model  and  procedure  for  using  it.  Finally  we  demon- 
strate the  proposed  comparison  methodology  through  comparative  evaluation 
of  the  predicted  relative  impact  of  two  subsets  of  TRW  MPP  in  the  context 
of  a hypothetical  software  development  activity.  Section  5 summarizes  and 
discusses  the  more  significant  study  conclusions,  describes  the  relation- 
ship of  TRW  MPP  to  current  Air  Force  practices,  and  provides  recommendations 
for  continued  research  and  increased  application  of  high-impact  MPP. 

In  summary,  TRW's  study  of  the  Impact  of  MPP  on  System  Development 
and  the  study  results  reported  here  represent  important  progress  toward 
confirmation  of  the  hypothesis,  viz: 

Modern  Programming  Practices,  if  rigorously  defined  and 
conscientiously  applied,  and  if  supported  by  appropriate 
techniques  and  tools,  help  make  it  possible  to  produce 
software  of  higher  than  usual  quality  at  lower  than  usual 
cost. 
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2.0  SYSTEMS  TECHNOLOGY  PROGRAM  (STP)  ENVIRONMENT 


2.1  General 

TRW  chose  STP  as  the  "guinea  pig"  for  studying  MPP  impact  for  several 
reasons.  First,  software  development  to  date  spans  almost  five  years  during 
which  many  modern  programming  practices  have  been  applied,  most  from  the 
beginning  but  some  during  only  more  recent  development  activity.  Second, 
a great  deal  of  data  has  been  collected  and  is  regularly  used  to  analyze 
and  evaluate  the  performance  of  STP  developers  and  the  software  they 
produce,  and,  based  upon  these  analyses,  an  effort  has  been  made  to  define 
new  production  practices  and  supply  developers  with  new  tools  to  improve 
performance. 

There  is  one  overriding  reason  for  the  large  number  of  MPP  used  in 
the  STP  development  activity.  At  the  outset  of  the  project  (formerly 
called  the  Site  Defense  (SD)  program),  there  was  considerable  feeling  out- 
side TRW  that  the  SD  data  processing  subsystem,  and  especially  the  soft- 
ware could  never  be  built  to  meet  the  demanding  performance  specifications, 
and  certainly  not  within  the  projected  cost  and  schedule.  To  meet  this 
challenge,  it  was  necessary  to  produce  not  only  the  heart  of  the  SD  system 
• (the  real  time  software)  but  also  the  development  support  software  and  test 

support  software  required  to  demonstrate  that  satisfactory  performance  had 
been  achieved.  The  sheer  magnitude  of  the  software  (nearly  1 million  machine 
instructions)  and  the  project  (as  many  as  400  people)  demanded  unprecedented 
rigor  in  the  development  process  and  led  to  the  establishment  and  enforced 
application  of  a variety  of  both  existing  practices  and  new  practices 
unique  to  STP. 

The  following  subsections  provide  a brief  description  and  historical 
account  of  STP  to  provide  the  reader  with  a general  understanding  of  the 
environment  within  which  these  modern  programming  practices  were  applied. 

2.2  System  Description 

Site  Defense  (SD)  was  intended  to  be  an  anti-bal 1 istic  missile 
terminal  defense  system.  It  was  designed  to  possess  a performance  credi- 
bility sufficient  to  deter  an  aggressor  from  a first-strike  attack  against 
the  Minuteman  (W)  force  and  to  ensure  that  an  acceptable  number  of  MMs 
survive  in  the  event  of  a first  strike.  Furthermore,  the  SD  System  would 
be  capable  of  countering  attacks  of  various  levels  and  tactics,  and  of 
degrading  gracefully  in  the  event  of  subsystem  overloads  or  failures,  or  in 
the  event  of  attacks  of  greater  severity  than  the  design  threat  parameters. 
The  SD  System  development  has  been  redefined  to  be  a BMD  Systems  Technology 
Program  (STP)  to  provide  objective  evidence  of  the  performance  of  the  key 
functions  of  a tactical  SD  System.  The  primary  objectives  of  STP  are: 

• Validating  the  data  processing  subsystem  by  demonstrating  the 
performance  of  the  engagement  software  (the  tactical  applica- 
tions program  (TAP)  and  the  tactical  operating  system  (TOS)) 

„ executed  by  the  computer,  a CDC  7700,  against  both  real  targets 

and  simulated  threats. 
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• Providing  the  framework  for  incorporating  currently  deferred 
data  processing  subsystem  elements. 

• Supporting  data  gathering  during  system  tests  at  Kwajalein 
missile  range  (KMR). 

The  STP  software  being  developed  by  TRW  has  been  organized  into 
engagement  software  (ESW),  test  support  software  (TSS),  and  development 
support  software  (DSS)  categories.  The  software  in  these  categories  con- 
sists of  ten  major  computer  programs  and  involves  a total  of  nearly 
1,000,000  machine  instructions. 

The  ESW  is  the  software  necessary  to  identify  and  track  ballistic 
re-entry  vehicles  through  the  use  of  STP  system  resources.  The  logic  and 
algorithms  within  the  ESW  consist  of  those  needed  to  satisfy  functional 
and  performance  requirements  which  support  the  system  engagement  func- 
tions: detect  and  designate  objects,  track  objects,  and  discriminate 
objects. 

The  software  that  actually  runs  on  the  CDC  7700  (the  STP  pro- 
cessor), is  referred  to  as  a process.  A process  is  composed  of  a data 
base,  an  operating  system,  and  one  or  more  application  programs.  The  Tacti- 
cal Application  Program  (TAP)  is  composed  of  tasks,  which  are  composed  of  sev- 
eral levels  of  routines.  The  T0S  is  a table-driven,  real-time  operating 
, system  designed  specifically  for  the  CDC  7700.  TAP  and  the  test  support 

programs  operate  under  its  control.  T0S  provides  the  following  basic  func- 
I tions:  task  supervision,  scheduling,  dispatching,  real-time  input  and 

output,  system  timing,  data  management,  history  logging,  error  detection, 
error  processing,  initialization,  and  termination. 

. The  primary  component  of  the  Test  Support  Software  (TSS)  is  the  KMR 

Test  Support  Program  (KTSP).  KTSP  is  required  to  test  key  functions  of  the 
engagement  process  and  to  support  system  test  operations  at  KMR.  Addition- 
al test  support  functions  have  been  provided  in  the  Data  Processing  Sub- 
system Simulator  (DPSS)  and  a variety  of  test  tools  used  in  the  genera- 
tion of  test  data  and  evaluation  of  test  results.  Development  of  two  other 
TSS  components  (i.e.,  the  System  Environment  and  Threat  Simulator  (SETS) 
and  the  System  Test  Driver  (STD))  was  initiated,  but  continued  development 
•j  was  recently  deferred. 

The  Development  Support  Software  consists  of  the  Basic  Operating 
System  (BOS),  the  Process  Construction  Program  (PCP),  specialized  develop- 
ment support  tools,  and  the  Data  Reduction  and  Report  Generator  (DRRG). 

BOS,  the  primary  operating  system  used  in  software  development,  consists 
of  a specially  tailored  version  of  the  SCOPE  2 operating  system  for  the 
CDC  "’600/7700  plus  the  associated  loaders,  compilers,  assemblers,  and 
utilities.  The  functions  of  PCP  are  data  base  definiton,  data  base  genera- 
tion, task/routine  compilation,  process  consolidation  and  process  adapta- 
b tion  using  a higher-level,  FORTRAN-like  language  which  facilitates 

construction  of  the  real-time  software.  PCP,  using  the  process  designer's 
input  directives  and  definitions  and  a library  file  of  coded  routines  and 
tasks,  assembles  the  application  program,  constructs  the  tables  defining 
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the  program  operation  to  the  tactical  operating  system,  and  links  the 
entire  process  together. 

Specialized  tools  consist  of  a number  of  utility  programs  that  aid 
developers  in  the  design,  execution,  and  evaluation  of  SD  software.  They 
provide  information  useful  in  static  and  dynamic  analysis  of  the  soft- 
ware and  relieve  developers  and  testors  of  many  repetitive  and  tedious 
tasks. 

DRRG  is  an  off-line  program  used  to  postprocess  in  a non-real -time 
environment  the  data  generated  by  the  test  processes.  DRRG  supports 
analysis  and  reduction  of  real-time  execution  history  logs  and  generates 
reports  based  on  user  requests. 

2.3  STP  Programming  Environment 

2.3.1  The  Computing  Facility 

The  major  elements  of  the  computer  hardware  system  consist  of 
CDC  6400,  7600,  and  7700  computers.  The  CDC  6400  computer  is  used  primar- 
ily as  an  input  and  output  station  for  the  CDC  7600  and  CDC  7700  computers. 
The  CDC  7600  computer  was  used  in  the  early  phase  of  software  development 
and  was  replaced  by  the  CDC  7700,  which  is  essentially  two  CDC  7600's 
with  a shared  large  core  memory.  The  7600  and  7700  main  frames  have  been  sup 
ported  by  an  extensive  complement  of  peripherals  driven  by  the  CDC  6400. 

At  the  peak  of  the  configuration  the  system  contained  two  card  readers, 
one  card  punch,  six  tape  drives,  eleven  disk  packs,  six  printers,  one  large 
disk  file,  and  three  operators  consoles.  Off-line  peripherals  consisted  of 
one  IBM  360-20  and  three  calcomp  plotters. 

All  STP  software  development  up  to  July  1976  has  been  done  in 
batch  mode  with  some  standalone  testing  of  the  real  time  system.  No  time 
sharing  or  remote  job  entry  systems  have  been  used.  After  the  CDC  7700 
system  was  shipped  to  KMR  (July  1976)  a remote  job  entry  (RJE)  system 
connected  to  the  McDonnell  Douglas  Astronautics  computer  facility  in 
Huntington  Beach,  California  has  been  used  by  TRW  for  software  maintenance 
and  additional  development.  At  the  peak  operation  (mid-1974)  approximately 
600  batch  jobs  per  day  were  processed  by  the  computer  facility  at  TRW. 
Availability  of  the  hardware  grew  from  85%  (in  early  1973)  to  an  average 
of  97%  (mid-1976).  Turn  around  time  was  greatest  (4  hours)  in  mid  1974  for 
classified  runs  and  has  averaged  approximately  40  minutes  for  all  runs  from 
mid  1974  through  mid  1976.  Throughout  1975,  the  second  7600  processor  was 
devoted  for  one  full  shift  to  real-time  testing  of  T0S  and  T0S/TAP.  The 
computer  facility  was  available  for  1.5  shifts  in  1973,  2.5  shifts  in  1974, 

2 shifts  in  1975,  and  two  shifts  for  the  first  6 months  of  1976. 

CDC  provided  a complete  set  of  user  documentation  for  the  computer 
system  and  vendor  supplied  software.  The  early  availability  of  up-to-date, 
accurate,  and  well  written  software  documentation  provided  programmers  who 
were  not  familiar  with  the  CDC  system,  a source  of  readi ,y  available 
documents  which  were  kept  current  by  CDC  through  their  documentation  update 
system.  Appendix  B contains  a list  of  the  major  vendor  supplied  software 
elements. 
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A systems  support  group  was  established  to  be  responsible  for 
maintaining  the  CDC  supplied  software.  This  group  had  the  responsibility  to 
develop  STP  unique  improvements  and  additions  to  the  CDC  software.  A major 
product  of  this  group  was  the  Basic  Operating  System  (BOS)  under  which 
most  STP  software  was  developed.  BOS  is  an  extension  of  the  SCOPE  2.1 
Operating  System  and  provides  special  features  which  are  required  for  STP 
software  development. 

2.3.2  The  People 

TRW's  experience  in  large  scale  software  systems  development, 
anti-bal 1 istic  missile  technology,  radar  systems,  and  systems  engineering 
provided  a readily  available  source  of  highly  qualified  software  managers, 
programmers,  systems  analysts,  software  engineers,  and  support  personnel 
for  assignment  to  the  STP  project.  The  large  TRW  software  personnel  pool 
and  the  STP  proposal  team  made  it  possible  to  staff  the  STP  project  with  a 
nucleus  of  qualified  people,  augmented  by  additional  people  hired  in  the 
open  job  market.  At  its  peak  the  project  staff  exceeded  400  personnel.  A 
history  of  the  project  population  through  mid-1976  is  shown  in  Figure  2-1. 
The  figure  also  illustrates  the  personnel  mix  (educational  background, 
degree  level  and  experience)  that  has  remained  relatively  constant 
despite  the  large  variation  in  project  size. 

The  initial  set  of  STP  software  managers  and  key  technical 
personnel  from  the  STP  proposal  team  were  familiar  with  the  detailed 
technical  issues  of  the  software  tasks,  and  enabled  the  STP  project  to 
operate  effectively  from  the  beginning  with  very  little  of  the  typical 
initial  start-up  "learning  curve"  problems.  In  fact,  all  of  the  key 
management  and  technical  positions  in  the  early  project  organization 
(Figure  2-2)  were  readily  filled  by  members  of  the  TRW  STP  proposal  team. 

As  the  STP  organization  changed  in  keeping  with  the  growth  in  project 
personnel  and  the  progressive  phases  of  development  activity,  the  approach 
to  software  management  has  been  to  promote  from  within  the  project  or  to 
use  experienced  TRW  personnel  from  other  projects  to  fill  key  positions. 

The  programmers  on  the  STP  project  fell  into  three  major 
categories:  1)  CDC  operating  systems  support  programmers,  2)  real  time 
systems  programmers,  and  3)  application  programmers.  Since  the  primary 
computing  facilities  at  TRW  already  use  CDC  6000  series  computers,  the  STP 
project  had  available  to  it  a source  of  qualified  systems  programmers  who 
needed  little  additional  training  to  use  the  CDC  7000  series  computers. 
Also,  since  the  Tactical  Application  Program  and  Test  Support  Software 
were  written  in  FORTRAN,  the  availability  of  experienced  programmers 
within  TRW  and  the  open  job  market  was  not  a problem.  TRW's  experience  in 
simulation,  radar  systems,  antiballistic  missile  technology,  test  tool 
development,  and  test  support  software  provided  a nucleus  for  the  applica- 
tions programming  area.  In  short,  obtaining  a qualified  STP  programming 
staff  was  not  a problem. 


The  STP  Program  began  with  the  award  of  multiple  contract  defini- 
tion (CD)  phase  contracts  to  three  teams  of  contractors  one  of  which  was 
the  team  of  McDonnell-Douglas,  TRW,  GE,  and  CDC.  At  the  end  of  the  CD  phase, 
each  team  was  required  to  submit  proposals  for  future  STP  development  and 
the  McDonnell-Douglas  team  won  this  competition.  Since  that  initial  award 
the  program  has  undergone  multiple  redefinitions  (here  called  "Reprogramming") 
mainly  as  the  result  of  congressional  action.  The  major  effect  of  this  re- 
programming has  been  to  stretch  schedules  and  redefine  the  content  of  the 
computer  program  deliverables.  The  approximate  contractual  chronology  for 
STP  is  as  follows: 


0 CD  Contract  Award  - April  1971 

• CD  Phase  Complete/STP  Proposal  Submitted  - November  1971 
0 STP  Contract  Award  - March  1972 

0 First  Reprogramming  (66  month  program)  - April  1973 

0 Second  Reprogramming  (72  month  program)  - March  1974 

0 Third  Reprogramming  (80  month  program)  - February  1975 
0 Fourth  Reprogramming  (86  month  program)  - March  1976 


Owing  largely  to  the  effect  of  the  reprogramming  on  the  STP  soft- 
ware development  schedules,  but  also  due  to  TRW's  phased  (Incremental 
Development)  approach,  it  is  extremely  difficult  to  provide  a single,  yet 
meaningful  illustration  of  the  detailed  STP  development  chronology.  For 
this  reason.  Figure  2-3  depicts  the  start  and  completion  dates  for  incre- 
ments (called  "loops"),  versions  and  critical  releases  of  major  ESW,  TSS  and 
DSS  components.  More  detailed  chronologies  are  provided  in  tabular  form  in 
Appendix  B. 
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3.0  MPP  IMPACT  EVALUATION  APPROACH  AND  RESULTS 


3.1  Detailed  Technical  Approach 

In  response  to  the  Statement  of  Work,  TRW  defined  and  followed  the 
MPP  study  approach  described  below  as  a sequence  of  tasks.  The  tasks,  in 
combination,  provided  necessary  engineering  analyses  to  1)  accomplish  an 
in-depth  evaluation  of  the  effectiveness  of  MPP  in  the  STP  software 
development  environment,  and  2)  develop  and  demonstrate  a methodology  for 
comparative  evaluation  of  alternative  MPP  implementations.  Figure  3-1 
illustrates  the  approximate  sequencing  of  project  activities  and  demonstrates 
the  high  interconnectivity  of  the  tasks  described  below. 

Task  1:  STP  MPP  Evaluation 

This  task  included:  identification  of  candidate  practices  and  selec- 
tion ana  aefinition  of  a final  set  of  practices  to  be  included  in  the  impact 
eva^ation  study;  collection,  organization  and  analysis  of  pertinent  STP  data 
anj  reconstruction  of  the  chronology  of  major  milestones  in  the  project 
development  cycle;  development  of  definitions  for  project  terminology; 
development  of  a description  of  the  STP  software  development  environment; 
and  detailed  analysis  to  determine  the  effects  of  adopting  specific  program- 
ming practices. 

Subtask  1.1:  MPP  Identification,  Selection  and  Definition 

This  subtask  included:  review  of  the  candidate  MPP  list  provided  in 
the  TRW  MPP  Study  proposal;  identification  of  additional  key  practices 
through  interviews  with  STP  personnel;  refinement  of  the  MPP  set  through 
a series  of  surveys  and  ranking  exercises  to  permit  concentrated  study  and 
impact  evaluation  of  a small  number  of  key  STP  practices;  and  preparation 
of  both  brief  and  expanded  definitions  of  each  selected  practice  to  sup- 
port the  impact  evaluation  activity. 


$ Subtask  1.2:  STP  Chronology  Reconstruction 

This  subtask  included:  identification,  collection  and  review  of 
relevant  historical  records,  briefing  materials,  meeting  reports,  progress 
and  status  reports  and  formal  software  development  plans;  identification 
of  key  STP  events;  and  preparation  of  a reconstructed  chronology  of  STP 
events  including  begin  and  end  dates  of  distinct  development  activities 
for  each  of  the  major  components  of  the  software,  intermediate  and  final 
milestones,  and  dates  marking  both  the  availability  and  first  application 
of  specific  programming  practices. 
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Figure  3-1.  MPP  Project  Activity  Network 
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Subtask  1.3:  STP  Terminology  Definition 

This  subtask  included:  identification  of  both  general  and  specific 
terms  unique  to  or  especially  important  in  the  STP  and/or  MPP  impact 
evaluation  context;  preparation  of  brief  definitions  of  all  such  terms;  and 
production  of  a glossary  of  selected  definitions  for  inclusion  in  this 
report. 

Subtask  1.4:  STP  Development  Environment  Description 

This  subtask  included:  collection  and  review  of  formal  system  and 
data  processing  subsystem  documentation  and  project  staffing  historical 
records;  and  preparation  of  a description  of  the  STP  software  development 
environment  including  1)  general  characteristics  of  the  computer  hardware 
configuration  and  operational  support  software  (e.g.,  operating  system, 
library  utilities,  compilers,  assemblers)  2)  the  availability  of  computing 
resources  through  batch,  remote  batch  and  time  sharing  services  at  various 
stages  of  the  software  development,  and  3)  an  illustration  of  project 
personnel  staffing  in  terms  of  the  availability  of  specialized  skills, 
fields  and  levels  of  education  and  directly  related  experience  during 
major  phases  of  the  software  development  activity. 

Subtask  1.5:  MPP  Impact  Evaluation 

This  subtask  included: 

t design  of  a framework  to  provide  for  the  collection  and  repre- 
sentation of  quantitative  data  on  the  impact  of  practices  on 
software  quality,  cost  and  schedule 

# development  of  a technique  ( i . e . , a modified  Delphi  exercise)  for 
bbtaining  individual  assessments  of  the  impact  of  practices  based 
on  detailed  analysis  of  available  data  by  key  STP  personnel 

0 preparation  for  and  performance  of  two  surveys  of  a cross  section 
of  STP  personnel  to  obtain  impact  assessments  of  1)  detailed 
Programming  Standards  on  desirable  software  qualities  and  2)  MPP 
on  typical  problems  that  have  plagued  software  development 

0 development  and  application  of  statistical  analysis  algorithms  to 
interpret  cumulative  response  data  and  formulate  composite  impact 
measures 

0 computation  of  figures  of  merit  representing  the  average  impact  of 
1)  each  practice  on  all  problems,  2)  all  practices  on  each  problem, 
and  3)  all  practices  on  all  problems. 
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Task  2:  MPP  Comparison  Methodology  Development 

This  task  included:  development  and  demonstration  of  a methodology 
for  comparative  evaluation  of  alternative  MPP  implementations;  and  prep- 
aration and  oral  presentation  of  a project  status  and  comparison  methodo- 
logy definition  briefing  to  RADC. 

Subtask  2.1:  Methodology  Definition 

This  subtask  included:  investigation  of  alternative  approaches  to 
MPP  comparison;  selection  of  a technique  that  was  both  1)  feasible  to 
implement  and  use  for  comparative  evaluation  of  many  MPP  implementations 
and  2)  compatible  with  the  MPP  impact  evaluation/representation  framework 
developed  in  Task  1;  and  demonstration  of  the  comparison  methodology  via 
comparative  evaluation  of  two  distinct  MPP  implementations. 

Subtask  2.2:  Oral  Presentation 

This  subtask  included:  collection,  analysis  and  refinement  of  mat- 
erials relevant  to  the  status  of  the  MPP  definition,  impact  evaluation  and 
comparison  methodology  development  activities;  and  preparation  and  presen- 
tation of  an  oral  briefing  to  RADC  personnel. 

Task  3:  Technical  Report  Preparation 

This  task  included  all  activities  necessary  for  preparation  of  this 
technical  report  to  include: 

t Information  to  describe  the  TRW  Systems  Technology  Program  (STP) 
software  development  activity  and  key  modern  programming 
practices  used  in  STP  software  production, 

• A description  of  the  approach  taken  to  obtain  quantified 
assessment  of  the  impact  of  MPP  on  software  quality,  cost  and 
schedule,  the  results  achieved  from  the  impact  evaluation 
studies,  and  corresponding  conclusions  and  recommendations, 

o A description  of  the  methodology  for  comparative  evaluation  of 
alternative  MPP  implementations  and  an  exemplary  demonstration 
of  the  methodology  and  support  tools  in  comparing  the  impact  of 
two  subsets  of  STP  MPP,  and 

• Rationale  and  recommendations  for  1)  Air  Force  consideration  of 
key  practices  for  inclusion  in  evolving  software  development 
guidelines,  and  2)  continued  research  and  development  tasks 
necessary  to  make  proper  use  of  TRW  MPP  study  results  and 
achieve  improved  insights  into  the  contribution  of  modem 
programming  practices  to  a disciplined  and  cost-effective  soft- 
ware engineering  process. 
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3 . 2 Selection  of  MPP 

The  term,  Modern  Progiamming  Practices  (MPP),  generally  refers  to 
standard  programming  practices,  software  design,  construction,  test  and 
documentation  techniques  and  management  methods  used  in  the  production  of 
software  for  the  purpose  of  enhancing  certain  qualities  of  the  software 
and/or  reducing,  eliminating  or  circumventing  known  software  production 
difficulties.  Clearly,  a review  of  almost  any  large  software  develop- 
ment activity  could  be  expected  to  uncover  some  practices  that  easily 
qualify  as  MPP.  Due  to  the  highly  critical  nature  of  the  STP  software 
development,  it  is  not  surprising  to  find  that  the  set  of  such  practices 
employed  at  various  stages  of  the  project  is  large  indeed.  Given  the 
primary  objective  of  the  MPP  study  (i.e.,  to  evaluate  and  quantify  the 
impact  of  MPP  on  the  STP  development)  and  given  limited  time  and  funding 
to  complete  the  task,  it  was  clearly  necessary  to  choose  a reasonably 
small  number  of  MPP  to  be  studied  so  that  sufficient  attention  could  be 
given  to  each. 

Our  approach  to  obtaining  a final  list  of  MPP  was  straightforward 
and  involved  the  following: 

• Identify  a baseline  set  of  practices  through  extensive  inter- 
views with  key  STP  personnel. 

t Survey  a cross  section  of  STP  and  non-STP  personnel  to  obtain  a 
relative-importance  ranking  of  the  MPP  and  to  identify  addi- 
tions to  the  baseline  set. 

• Analyze  the  expanded  list  of  MPP,  delete  those  of  low  rank  and 
those  highly  peculiar  to  STP  and,  where  appropriate,  combine 
several  related  individual  practices  into  one  more  general 
practice. 

Through  iterative  application  cf  the  above  steps,  we  progressed  from  the 
original  list  of  14  candidate  practices  to  the  final  set  of  11  MPP  as 
illustrated  by  Figure  3-2.  Appendix  A contains  brief  descriptions  of  the 
14  candidate  practices  and  detailed  definitions  of  the  final  11  MPP. 

It  is  important  to  note  that  we  sought  not  only  a manageable  number 
of  MPP  upon  which  to  concentrate  the  impact  analyses  but  also  a representa- 
tive collection  of  practices  that  spanned  the  major  phases  of  STP  develop- 
ment. Of  the  set  of  11  MPP: 

• 4 practices  apply  mainly  to  the  requirements  definition/pre- 
liminary design  phases  of  software  development: 

- Requirement  Analysis  and  Validation 

- Baselining  of  Requirements  Specification 

- Complete  Preliminary  Design 

- Process  Design 
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• 3 practices  apply  mainly  to  the  detail  design  and  coding  phases: 

- Enforced  Programming  Standards 

- Incremental  Development 

- Unit  Development  Folders 

• 2 practices  apply  mainly  to  the  test  phase: 

- Independent  Testing 

- Formal  Inspection  of  Documentation  and  Code 

• 2 practices  apply  to  all  phases  of  software  development: 

- Software  Development  Tools 

- Software  Configuration  Management 

3 . 3 Impact  Evaluation  Study 

A primary  objective  of  the  Impact  of  MPP  on  System  Development  pro- 
ject was  to  obtain  a quantified  assessment  of  the  nature  and  extent  of  the 
effect  of  modern  practices  applied  to  the  TRW  STP  software  development. 

Doing  sc  was  expected  to  provide  important  evidence  with  respect  to  the 
general  hypothesis  (Section  1.2)  as  to  the  net  positive  impact  of  MPP  on 
software  quality  and  cost. 

Selection  of  an  impact  assessment  approach  and  methodology  was 
strongly  influenced  by  previous  TRW  experience  on  several  related  advanced 
software  technology  studies.  These  were  the  Quantitative  Software  Safety 
Study  l_14J  under  contract  to  McDonnell  Douglas  Astronautics  Company  and  the 
Characteristics  of  Quality  Software  study  [l5]  for  the  National  Bureau  of 
Standards.  Both  of  these  studies  involved  measuring  the  quality  of 
software  and  required  identification  of  explicit  characteristics  (i.e., 
detailed  qualities)of  software  and  associated  metrics  with  which  a 
quantified  quality  assessment  could  be  achieved. 

Similarly  for  the  MPP  study,  the  approach  to  evaluating  the  overall 
impact  of  practices  on  software  quality  and  cost  involved  1)  establishment 
of  a common  set  of  software  characteristics,  and  2)  development  and  use  of 
a technique  to  obtain  measures  of  the  nature  and  extent  of  the  cause  effect 
relationship  between  each  practice  ar.d  each  characteristic. 

In  view  of  particular  interest  in  programming  standards,  two  distinct 
impact  evaluation  activities  were  planned  and  conducted.  The  first  focused 
on  18  detailed  documentation  and  coding  standards  from  the  current  version 
of  the  STP  Software  Standards  and  Procedures  Manual  Lio]  and  sought  an  impact 
assessment  relative  to  30  characteristics  of  software  and  the  software 
development  process.  The  second  impact  evaluation  activity  then  concen- 
trated on  the  11  more  general  MPP  and  sought  an  impact  assessment  relative 
to  12  typical  problems  that  have  plagued  (and  all  too  often  characterize) 
software  development.  The  first  evaluation  thus  called  for  540  (i.e.,  18  x 30) 
individual  measures  of  the  impact  of  the  practices  on  the  characteristics, 
while  the  second  required  132  (i.e.,  11  x 12)  impact  measures,  one  for  each 
practice-problem  combination. 
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Owing  to  the  large  number  of  individual  impact  measures  to  be  ob- 
tained and  the  attendant  magnitude  of  effort  required  to  collect  and 
evaluate  data  for  each,  it  was  decided  that  the  basic  technique  for  obtain- 
ing impact  measures  should  be  a modified  Delphi  exercise  involving  several 
surveys  of  experienced  TRW  personnel.  Use  of  this  technique  not  only  made 
the  acquisition  of  comprehensive,  detailed  impact  assessment  data  a practi- 
cal task,  it  also  brought  into  the  impact  evaluation  activities  a broad 
spectrum  of  project  knowledge  and  thus  varying  impact  assessments  based  on 
the  unique  and  differing  experiences  of  a cross  section  of  STP  personnel. 

In  order  to  ensure  maximum  objectivity  in  the  analysis  of  response  data 
and  formulation  of  a composite  impact  measure  for  each  practice-character- 
istic combination,  analysis  algorithms  were  established  in  advance  and 
implemented  in  the  RANK,  IMPACT  and  MERIT  support  programs. (See  Appendices  C 
and  D for  detailed  descriptions  of  the  algorithms  and  support  tools.) 

In  essence,  the  impact  evaluation  approach  was: 

• Conduct  a survey  requiring  each  participant  to  provide  an 
individual  measure  (influence  rating)  of  the  impact  of  specific 
practices  on  certain  characteristics  of  software  development 

• Analyze  the  survey  response  data  as  necessary  to  obtain  a 
composite  impact  measure  for  each  practice-characteristic 
combination  as  well  as  measures  of  the  impact  of  1)  each  practice 
on  all  of  the  characteristics,  2)  all  of  the  practices  on  each 
characteristic,  and  3)  all  of  the  practices  on  all  of  the 
characteristics. 

3.3.1  Programming  Standards  Impact 

The  first  survey  concentrated  on  detailed  programming  standards  and 
sought  measures  of  their  impact  on  characteristics  of  software  and  the 
software  development  process.  The  survey  questionnaire  was  completed  by  54 
people  representing  a cross  section  of  STP  personnel  as  follows: 


Tactical 
Software 
Design  & 
Development 

Operating 

System 

Software 

Test  Support 
Software 
Design  & 
Development 

Software 

Integration 

and 

Test 

Product 
Assurance  & 
Configuration 
Management 

Other 

17 

9 

11 

11 

3 

3 

The  Programming  Standards  Impact  Evaluation  survey  questionnaire 
contained  three  parts: 


1)  A two  dimensional  matrix  of  18  programming  standards  versus  30 
software  characteristics  ( i . e . , 540  "multiple  choice/fi 1 1 -in- 
the-blank"  questions) 

2)  A true/false  question  posed  as  a restatement  of  the  general  MPP 
hypothesis:  "Programming  (i.e.,  documentation  and  coding)  stan- 
ards,  if  rigorously  defined  and  systematically  enforced  with 
support  tools,  help  make  possible  the  production  of  software  of 
higher  than  usual  qual i ty  at  lower  than  usual  cost". 

3)  A request  for  a free-form  statement  (in  25  words  or  less)  as  to 
the  general  effect  of  programming  standards  on  STP. 
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Figure  3-3  illustrates  the  blank  survey  questionnaire. 

Each  survey  participant  was  asked  to  complete  as  much  of  the 
matrix  as  possible  and,  from  personal  experience,  to  indicate  both  the 
nature  and  amount  of  influence  of  each  standard  on  each  characteristic 
by  choosing  and  entering  the  most  appropriate  influence  rating  from  the 
following : 

+2  Strong  Positive  Influence 

+1  Some  Positive  Influence 

0 No  Influence 

-1  Some  Negative  Influence 

-2  Strong  Negative  Influence 

Blank  Unknown  Influence 


Upon  completion  of  the  survey  the  mathematical  algorithms  and 
support  programs  were  used  to  analyze  and  summarize  the  response  data. 

The  IMPACT  program  provided  analysis  of  the  response-frequency-  distri- 
bution (i.e.,  the  number  of  +2,  +1,  0,  -1,  and  -2  responses)  for  each 
element  of  the  matrix,  computation  of  mean  and  variance,  and  application 
of  a hypothesis  testing  procedure  (summarized  in  Table  C-III  of  Appendix  C) 
to  derive  both  summary  assessments  of  impact  (i.e.,  positive,  negative  or 
none)  and  confidence  levels  (i.e.,  strong,  medium,  weak)  for  assertions: 

"Practice  i has  a positive  (negative)  influence  on  characteristic  j." 

The  summarized  results  of  the  Programming  Standards  Impact  Evaluation 
survey  are  presented  in  Figure  3-4.  The  figure  also  contains  a legend  de- 
scribing the  abbreviated  notation  for  the  possible  combinations  of  assertion 
strength/influence  rating  which  appear  in  the  elements  of  the  matrix,  an 
overall  summary  of  the  matrix  contents,  and  the  percentages  of  true,  false 
and  "don't  know"  responses  to  the  general  hypothesis. 

The  IMPACT  program  also  computed  influence  ratings  for  each  row 
(practice),  each  column  (characteristic),  and  for  the  total  matrix  corre- 
sponding to  1)  the  average  effect  of  each  practice  on  the  combined  set  of 
characteristics,  2)  the  average  effect  of  the  combined  practices  on  each 
characteristic,  and  3)  the  average  effect  of  the  combined  practices  on  the 
combined  set  of  characteristics.  These  additional  assertion  strength/in- 
fluence ratings  can  be  viewed  as  a 19th  row  and  31st  column  of  the  matrix 
as  indicated  in  Figures  3-5  and  3-6. 

The  assertion  strength/influence  rating  for  the  combined  practices 
on  the  combined  characteristics  was  determined  to  be  Weak/Positive.  The 
weakness  of  the  assertion  strength  is  due  to  the  large  number  (130)  of 
Indifferent  (i.e.,  no  effect)  ratings  for  individual  practice-characteristic 
pairs  , while  the  overall  influence  rating  derives  from  the  ratio  of  positive 
to  negative  influence  ratings  (292/17  * 17.2)  which  clearly  suggests  Positive 
impact. 


— — 
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Figure  3-3.  Programming  Standards  Survey  Questionnaire 
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Practice  Assertion  Strength/ 

Index  Practice  Identification  Influence  Rating 


1 

Text  Format 

I 

WP 

2 

Text  Level  of  Detail 

MP 

3 

Flow  Chart  Format 

WP 

4 

Flow  Chart  Level  of  Detail 

MP 

5 

Statement  Label  Format 

WP 

6 

Executable  Statement  Format 

WP 

7 

Routine  Size  (Modularity) 

MI 

8 

Calling  Sequence  Arguments 

WP 

9 

Mixed  Mode  Arithmetic 

WI 

10 

Do-LOOP  Usage 

WP 

11 

Computed  G0-T0  Usage 

WI 

12 

Labeled  Common  vs  Blank  Common  Usage 

WP 

13 

Imbedded  Physical  Constants  Usage 

WP 

14 

Preface  Commentary  Block  Requirement 

MP 

15 

In-line  Commentary  Requirement 

MP 

16 

Structured  Coding  Requirement 

MI 

17 

Execution  of  Every  Program  Branch  Requirement 

WI 

18 

Naming  Conventions 

WP 

Figure  3-5.  • Practice  Impact  on  Combined  Characteristics 
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Characteristic  Characteristic  Identification  Assertion  Strength/ 

Index Influence  Rating 


1 

Requirements  Traceability 

WP 

2 

Code  Auditability 

MP 

3 

Documentation  Auditability 

WP 

4 ! 

Personnel  Motivation 

WI 

5 

Interface  Consistency 

WP 

6 

Software  Integration 

WP 

7 

Documentation  Rework 

WI 

8 

Coding  Rework 

MI 

9 

Retesting 

WP 

10 

Customer/Contractor  Relationship 

MP 

11 

Testing  Thoroughness 

WP 

12 

Documentation  Under standabi 1 i ty/Readabi 1 i ty 

WP 

13 

Code  Understandabi 1 i ty/Readabi 1 i ty 

MP 

14 

Documentation  Mai ntai nabi 1 i ty/Usabi 1 i ty 

WP 

15 

Code  Ma i ntai nabi 1 i ty/Usabi 1 i ty 

MP 

16 

Testability 

WP 

17 

Operational  Reliability 

WP 

18 

Cost 

MI 

19 

Schedule 

WI 

20 

Consistency 

MP 

21 

Completeness 

WP 

22 

Documentabi 1 ity 

WP 

23 

Codability  (ease  of  coding) 

MI 

24 

Documentation  Error  Frequency 

WI 

25 

Coding  Error  Frequency 

MP 

26 

Portability  (Computer  Independence) 

WP 

27 

Execution  Time 

SI 

28 

Core  Requirements 

SI 

29 

Logical  Organization 

WP 

30 

Programmer  Productivity 

MI 

* 

"Row  19" 


Figure  3-6.  Combined  Practices  Impact  on  Each  Characteristic 
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There  are  far  too  many  individual  elements  of  the  matrix  to  permit 
detailed  discussion  of  the  assessed  impact  of  each  standard  programming 
practice  on  each  of  the  software  characteristics.  Certain  of  the  impact 
evaluation  results  are  particularly  interesting,  however,  and  are  deserving 
of  special  mention  and,  in  some  cases,  further  explanation.  These  are: 

• The  Structured  Coding  Requirement  received  a positive  influence 
rating  for  18  of  the  30  software  characteristics,  a negative 
for  7 characteristics,  an  indifferent  rating  for  one  character- 
istic, and  an  inconclusive  rating  for  all  the  rest,  including 
the  average  rating  across  all  30  software  characteristics.  Of 
the  18  positive  ratings,  however,  only  4 can  be  asserted  with 
strong  confidence,  5 with  medium  confidence  and  9 with  only 
weak  confidence.  On  the  other  hand,  just  one  of  the  7 negative 
ratings  can  be  asserted  with  medium  confidence  and  the  other  6 
with  strong  confidence.  Compared  with  the  other  standards  which 
all  together  received  only  10  negative  ratings,  this  indicates 

a relatively  dim  view  of  structured  coding  on  the  part  of  STP 
personnel.  There  are  some  good  reasons  for  this  to  be  the  case. 
First,  it  is  well  recognized  that  structured  coding  (unlike  most 
of  the  other  standards)  generally  requires  more  core  and  execution 
time.  Additionally,  the  standard  was  not  explicitly  defined,  estab- 
lished and  enforced  until  almost  two  years  after  STP  began,  and  the 
requirement  to  restructure  existing,  working  code  was  felt  to  be 
both  unnecessary  and  counter-producti ve.  Moreover,  writing  struc- 
tured code  in  standard  FORTRAN  is  awkward  and  introduced  some 
inefficiencies  in  core  and  execution  time  making  it  difficult 
to  satisfy  demanding  design  and  performance  requirements  levied 
on  the  real-time  software.  Finally,  STP  has  not  yet  reached  the 
phase  during  which  the  major  benefits  of  structured  programming 
(i.e.,  improved  readability  and  maintainability)  are  expected  to 
be  reaped. 

• Approximately  one  out  of  every  four  of  the  standard-characteristic 
pairs  received  an  assertion  strength/influence  rating  of  Indiff- 
erent. The  Indifferent  rating  implies  a strong  assertion  of  no 
impact,  in  that  over  75%  of  the  survey  participants  had  to  enter 

a no-influence  (0)  rating  for  the  composite  rating  of  Indifferent 
to  be  obtained.  Although  at  first  the  number  of  Indifferent 
ratings  seems  high,  careful  review  of  the  matrix  reveals  a very 
good  match  with  intuition. 
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There  are  eight  software  characteristics  which  received  no  nega- 
tive or  inconclusive  impact  assessments  for  any  of  the  18  stan- 
dards. These  are:  Code  Auditability,  Documentation  Auditability, 
Customer/Contractor  Relationship,  Documentation  Understandability/ 
Readability,  Code  Understandability/Readability,  Operational 
Reliability,  Consistency,  and  Completeness.  On  the  other  hand, 
Personnel  Motivation  received  14  inconclusive  ratings  (showing 
mixed  positive  and  negative  responses  from  the  survey  partici- 
pants), three  indifferent  ratings  and  one  negative.  It  appears 
that  the  overall  impact  of  standards  on  Personnel  Motivation 
correlates  much  better  with  the  preceived  impact  on  Cost  and 
Schedule  (i.e.,  only  two  positive  influence  ratings  for  each) 
than  with  the  above-mentioned  eight  characteristics.  This  says 
something  quite  positive  about  the  objectivity  of  the  responses; 
i.e.,  in  general  the  survey  participants  did  not  let  their  per- 
sonal distaste  for  a standard  adversely  effect  their  judgement 
of  its  effectiveness. 


The  results  for  the  Routine  Size  (Modularity)  programming  standard 
are  particularly  interesting  in  that  they  compare  quite  well  with 
those  of  Hoskyns  in  "Evaluation  of  Programming  and  Systems 
Techniques;  Implications  of  Using  Modular  Programming."  [l6] 

In  particular,  Hoskyns  reports  that  89%  of  those  surveyed  ex- 
perienced "easier  maintenance"  from  modular  programming.  By 
comparison,  the  software  characteristics  most  closely  related  to 
maintenance  (i.e.,  Code  Auditability,  Code  Understandability/ 
Readability,  and  Code  Maintainability/Usability)  all  received 
Strong/Positive  ratings.  Furthermore,  Hoskyns'  results  on 
"better  program  design"  and  "easier  program  testing"  (85%  and 
78%,  respectively)  compare  well  with  ours  for  Logical  Organiza- 
tion and  Testability,  both  of  which  received  Strong/Positive 
ratings.  Notably,  fewer  of  the  modular  programming  users  (64%) 
experienced  the  benefit  of  "more  reliable  programs",  again  com- 
paring well  with  our  results  showing  Medium/Positive  ratings  for 
both  Operational  Reliability  and  Coding  Error  Frequency.  On  the 
negative  side,  Hoskyns  reports  that  28%  of  the  users  claimed 
"more  computer  time  required  during  development"  and  27%  ex- 
perienced "less  efficient  final  programs",  corresponding  to  our 
Strong/Negative  and  Medium/Negative  results  for  Execution  Time 
and  Core  Requirements.  The  only  obvious  inconsistencies  between 
Hoskyns'  results  and  those  reported  here  are  for  Programmer 
Productivity  and  Schedule.  Hoskyns'  results  for  "higher  pro- 
grammer productivity"  (64%)  and  "more  likely  to  meet  target  dates" 
(63%)  are  based,  however,  on  a somewhat  relaxed  interpretation  of 
modularity  (i.e.,  20  to  2000  statements),  while  our  Strong/ 
Inconclusive  ratings  relate  to  a formal  standard  restricting 
module  size  to  a maximum  of  100  executable  FORTRAN  statements. 


mm* 
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To  complete  analysis  of  the  survey  responses,  the  free  form  responses 
were  carefully  examined.  From  review  of  all  54  responses,  a list  of  general 
response  categories  was  prepared  and  frequencies  were  derived  fnr  the  number 
of  times  each  response  category  appeared  in  the  total  set  of  responses.  Most 
of  the  free  form  responses  contained  two  or  more  of  the  response  categories 
(some  as  many  as  four).  Thus,  the  sum  of  the  category-appearance-frequencies 
is  much  greater  than  5*4,  the  number  of  free  form  responses  analyzed. 


Frequency  of 
Response  Category 
Appearance 


Response  Category 


Code  easier  to  understand,  maintain, 
test,  or  document,  or  better 
documentation 

Consistent  code  format 

Reduced  bugs  in  code,  more  reliable, 
higher  quality 

Guided  programmers,  guidelines  for  devel- 
opment 

Standards  introduced  late,  changing 
standards  caused  negative  impact 

Constrained  programmers'  creativity 

Immediate  cost  high,  schedule  impact 
negati ve 

Programmer  gripes,  disgruntlement 

Recognition  of  overall  cost  decrease 

Generally  positive  (unspecific) 

Disciplined  programmers 

Arbitrary  or  inconsistent  standards 

Not  applicable,  no  opinion,  or  wait  and 
see 


Quality  of  standards  insufficient,  could 
be  improved 

Interfaces  better,  easier 
Job  done  faster  and  better 
Generally  negative  (unspecific) 
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It  can  be  seen  from  this  view  of  the  free  form  responses  that  there  were 
almost  twice  as  many  positive  comments  (67)  as  there  were  comments  of  a 
negative  nature  (35).  Finally,  note  that  the  highest  frequency  categories 
represent  responses  generally  consistent  with  those  of  the  Impact  Evalua- 
tion, i.e.,  code  and  documentation  understandabi 1 i ty  and  maintainability, 
coding  error  frequency,  and  consistency  are  scored  generally  "positive". 
However,  the  categories  dealing  with  the  effect  on  programmers  appear  to 
indicate  a stronger  impact  than  does  the  Impact  Evaluation.  In  the  latter, 
personnel  motivation  and  programmer  productivity  did  not  seem  to  be  big 
issues,  except  for  the  effect  of  the  structured  coding  requirement.  This 
particular  standard  clearly  seemed  to  be  the  most  controversial  and  moti- 
vated many  of  the  free  responses. 

3.3.2  MPP  Impact 

The  second  impact  evaluation  survey  focused  on  the  11  previously 
identified  MPP  and  sought  an  assessment  of  the  extent  to  which  the  practices 
did  (in  the  case  of  STP)  and  could  (in  a hypothetical  software  development) 
contribute  to  elimination  of  typical  problems  that  have  plagued  and  adversely 
affected  software  development  activities.  The  survey  questionnaire  was 
completed  by  67  people  from  a cross  section  of  STP  personnel.  Of  the  67 
survey  participants,  31  held  management  positions  with  varying  levels  of 
project  responsibility. 

The  MPP  Impact  Evaluation  survey  questionnaire  was  designed  as  a 
booklet  containing  five  parts: 

1)  A series  of  individual  impact  rating  matrices,  one  for  each  of 
the  11  MPP.  Each  page  contained  a brief  definition  of  the  prac- 
tice and  a blank  12  x 5 evaluation  matrix.  Each  row  of  the  mat- 
rix corresponded  to  one  of  the  12  typical  development  problems, 
while  the  columns  offered  a choice  of  five  influence  ratings. 
Survey  participants  were  required  to  enter  an  "A"  in  the  approi- 
priate  column  of  each  row  to  indicate  the  nature  and  extent  of 
the  influence  of  the  practice  on  each  problem  in  the  actual  STP 
context,  and  to  enter  a "T"  to  indicate  the  theoretical,  optimum 
impact  of  the  practice  under  ideal  conditions.  Figure  5-7 
illustrates  the  impact  rating  matrix  used  to  evaluate  each  of  the 
MPP. 

2)  A list  of  the  11  practices  and  a request  that  they  be  ranked 
according  to  relative  importance  with  respect  to  their  positive 
contribution  to  elimination  of  software  development  problems. 

3)  A true/false  question  posed  as  a restatement  of  the  general  MPP 
hypothesis. 

4)  A list  of  the  12  typical  software  development  problems  and  a re- 
quest that  they  be  ranked  according  to  relative  significance 
with  respect  to  their  negative  effect  on  quality,  schedule  and 
cost  of  software  production. 

(The  questionnaire  format  for  items  2,3  and  4 is  illustrated  in 
Figure  3-8). 


MPP 

RATING 


HIGH 

POSITIVE 


PROBLEM 


COST  OVERRUN 


DEVELOPMENT  STATUS  INVISIBILITY 


UNRELIABILITY 


UN MAINTAINABILITY 


NO 

LOW 

EFFECT 

NEGATIVE 

INADEQUATE  SATISFACTION  OF  REAL 
REQUIREMENTS 


INEFFICIENT  USE  OF  RESOURCES 


SCHEDULE  OVERRUN 


INADEQUATE  PLANNING  AND  CONTROL 


PROJECT  MISMANAGEMENT 


LACK  OF  PROGRAMMING  DISCIPLINE 


LACK  OF  CONCLUSIVE  TESTING 


POOR  DOCUMENTATION 


Figure  3-7.  MPP  Survey  Questionnaire  Sample  Rating  Sheet 
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MPT  Ranking 


MPP 

RANK* 

Requirements  Analysis  end  Validation 

Process  Design 

YncrerrenYaT  Development  ' ' 

Complete  Prel fminary  Ue s i g n 

Unit  Development  Folders 

Enforced  Programming  Standards 

Independent  testing 

.Software  Configuration  Management 

Baselining  of  Requirements  Specification 

Formal  Inspection  of  Dccumentaticn  and  Code 

Software  Development  Tools  | 

General  MPP  Hypothesis:  Rules  governing  software  development,  evaluation, 

and  documentation,  if  rigorously  defined  and  applied,  and  supported  by 
modern  programming  practices  (techniques  and  tools),  make  possible  the 
production: 


True 

False 

? 

• of  higher  than  usual  quality  software 

□ 

□ 

□ 

• at  lower  than  usual  life  cycle  cost 

□ 

cn 

□ 

Soft ware  Development  Problem  Ranking 


PROBLEM 

RANK** 

Cost  Overrun 

Development  Status  Invisibility 

Unrel iabil ity 

Unnta  in  ta  inab  i 1 i ty 

Inadequate  Satisfaction  of  Real  Requirements 

Inefficient  Use  of  Resources 

Schedule  Overrun 

Inadequate  Planninq  and  Control 

Project  Mismanaaement 

Lack  of  Proqramm  nq  Discipline 

Lack  of  Conclusive  Testing 

Poor  Documentation 

* Relative  importance  ranking  ; use  rank  of  1 for  "most  important,"  2 for 

"next  most  important,"...  11  forecast 
important." 

**  Relative  significance  ranking:  Use  rank  of  1 for  "most  significant",  2 for 

"next  most  significant,"...  12  for"least 
significant." 


figure  3-8.  MPP  Survey  Questionnaire  Ranking  Sheet 
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5)  A request  for  a brief  free-form  statement  as  to  the  need  for 
and  benefits  derived  from  MPP  as  applied  to  STP  and  in  general. 

Analysis  of  the  MPP  Impact  Evaluation  survey  response  data  was 
accomplished  with  the  same  mathematical  algorithms  and  support  tools  used 
in  the  programming  standards  study.  The  IMPACT  program  (Appendix  D)was  used 
to  compute  influence  rating  averages  and  frequencies  and  to  apply  the 
hypothesis  testing  procedure  to  the  response-frequency-distibution 
for  each  practice-problem  pair.  The  hypothesis  testing  was  applied  inde- 
pendently to  both  the  actual  (A)  and  theoretical  (T)  responses,  yielding 
two  composite  assertion  strength/influence  rating  indicators  for  each 
practice-problem  pair  as  illustrated  in  the  MPP  Impact  Evaluation  survey 
summary  matrix  (Figure  3-9).  IMPACT  also  computed  an  average  response- 
frequency-distribution  for  each  of  the  11  rows  and  12  columns  and  for  the 
complete  matrix,  thus  providing  an  indication  of  the  cumulative  (average) 
impact  of  1)  each  practice  on  the  combined  set  of  problems,  2)  the  combined 
set  of  practices  on  each  problem,  and  3)  all  the  practices  on  all  the 
problems.  The  legend  in  the  upper  left  corner  of  the  figure  indicates  the 
appropriate  interpretation  of  the  matrix  contents.  For  example  at  the  inter- 
section of  row  9 and  column  5 there  is  an  SP  above  the  diagonal  and  an  SSP 
below  from  which  we  obtain: 

"We  can  assert  with  strong  confidence  that,  in  the  actual  case  of 
the  System  Technology  Program,  the  practice  (Baselining  of  Re- 
quirements Specification)  made  a positive  contribution  toward 
eliminating  the  problem  (Inadequate  Satisfaction  of  Real  Re- 
quirements)." 

"We  can  assert  with  strong  confidence  that,  in  the  theoretical 
case,  the  practice  (Baselining  of  Requirements  Specification) 
can  and  should  make  a strong  positive  contribution  toward  elimina- 
ting the  problem  (Inadequate  Satisfaction  of  Real  Requirements)." 

The  Survey  Summary  on  the  right  side  of  the  figure  indicates  the 
number  of  times  each  assertion  strength/influence  rating  appears  in  the 
matrix,  excluding  the  12^  row  and  13^  column.  For  example,  the  rating 
SP  (i.e.,  strong  confidence/positive  impact)  was  obtained  for  24  practice- 
problem  combinations  in  the  actual  STP  case  and  for  54  combinations  in  the 
theoretical  case. 


The  figure  also  illustrates  the  composite  survey  response  to  the 
general  MPP  hypothesis  true/false  question,  indicating  that  85%  of  the 
survey  participants  agreed  that  MPP  contribute  to  the  production  of  "higher 
than  usual  quality  software."  On  the  issue  of  "lower  than  usual  life  cycle 
cost",  agreement  as  to  a positive  MPP  contribution  is  not  nearly  as  marked, 
however,  the  true  responses  outnumber  the  false  responses  by  almost  4 to  1. 

In  the  formulation  of  the  average  impact  of  each  practice  on  all 
problems  (i.e.,  the  assertion  strength/influence  ratings  in  column  13  of 
the  matrix),  equal  significance  of  all  problems  was  assumed  . From  the 
cumulative  response-frequency-distributions  for  all  of  the  practices,  it 
was  thus  possible  to  rank  the  practices  in  order  of  their  relative  impact 
on  the  set  of  uniformly  weighted  problems.  For  the  purposes  of  verifying 
the  survey  results  and  further  analyzing  and  estimating  the  relative  value 
of  the  practices  in  differing  development  activities,  several  other  rankings 
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Figure  3-9.  Mpp  Impact  Evaluation  Survey  Results 
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were  obtained.  The  RANK  program  (Appendix  D)  was  used  to  average  the  re- 
lative-importance-practice rankings  provided  by  the  survey  participants 
and  produce  a composite  rank  ordering  of  the  11  MPP  with  a mean  rank  and 
standard  deviation  for  each.  Similarly,  RANK  produced  a composite  relative- 
significance-problem  ranking  (with  means  and  standard  deviations)  using  as 
inputs  the  individual  rankings  from  the  67  survey  questionnaires.  The  mean 
ranks  for  the  problems  were  used  to  derive  the  problem  weights  shown  in 
Figure  3-10.  Using  these  weights  to  appropriately  factor  each  practice- 
problem  response-frequency-distribution,  the  MERIT  program  (Appendix  D) 
computed  an  average  distribution  for  each  practice  on  the  set  of  weighted 
problems.  MERIT  also  applied  the  statistical  hypothesis  test  to  the  response- 
frequency-distributions  for  the  theoretical  case,  producing  the  assertion 
strength/influence  ratings  and  the  relative  impact  ranking  for  the  practices 
in  which  the  varying  significance  of  software  production  problems  is  re- 
flected. The  results  of  the  three  separate  rankings  of  the  practices  are 
shown  in  Figure  3-11 . Notice  that,  as  a result  of  weighting  the  problems, 
the  assertion  strength/influence  rating  for  Baselining  of  Requirements 
Specification  changed  from  Medium/Positive  to  Strong/Positive  and  the 
relative  impact  rank  changed  from  4 to  2.  That  is,  in  a software  develop- 
ment activity  for  which  the  12  typical  problems  are  of  equal  (or  very 
\ nearly  equal)  significance,  it  can  asserted  with  medium  conTTdence  that: 

"The  practice,  Baselining  of  Requirements  Specification,  makes  a 
positive  contribution  toward  elimination  of  the  problems,  and 
only  2 other  practices  have  more  positive  impact." 

On  the  other  hand,  the  Strong/Positive  rating  and  the  rank  of  2 
imply  that,  in  a software  development  activity  for  which  the  problems  have 
varying  significance  (and  are  assigned  the  weights  of  Figure  3-10),  it  can 
be  asserted  with  strong  confidence  that: 

"The  practice,  Baselining  of  Requirements  Specification,  makes  a 
positive  contribution  toward  elimination  of  the  problems,  and 
only  1_  other  practice  has  more  positive  impact." 

Analysis  of  the  MPP  impact  evaluation  survey  summary  results  matrix, 
together  with  the  above  three  practice  rankings,  prompts  the  following 
general  observations: 

• There  is  strong  agreement  among  STP  personnel  as  to  the  four 
MPP  of  greatest  importance  and  impact  and  strong  agreement  on 
their  relative  ranking: 

- Requirements  Analysis  and  Validation 

- Baselining  of  Requirements  Specification 

- Complete  Preliminary  Design 

- Process  Design 

• There  is  strong  agreement  on  the  importance  and  positive  impact 
of  the  next  three  most  highly  ranked  MPP,  but  the  relative  rank- 
ing among  them  is  less  clear: 

- Incremental  Development 

- Unit  Development  Folders 


Weights 


Figure  3-11.  Comparison  of  Practice  Rankings 
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- Software  Development  Tools 

• There  is  strong  agreement  on  the  importance  and  positive  impact 
of  the  four  lower  ranked  MPP,  but  the  relative  ranking  among 
them  is  not  at  al 1 clear: 

- Independent  Testing 

- Enforced  Programming  Standards 

- Software  Configuration  Management 

- Formal  Inspection  of  Documentation  and  Code 


Detailed  analysis  of  the  67  free  form  MPP  survey  responses  address- 
ing the  need  for  and  benefit  of  MPP  on  STP  and  in  general  resulted  in 
identification  of  12  response  categories.  Many  of  the  free  form  responses 
fell  into  more  than  one  category.  The  following  describes  the  response  cate- 
gories and  identifies  the  number  of  comments  in  the  67  free  form  responses 
which  fell  into  each  category. 

Frequency  of 
Response  Category 

Appearance Response  Category 

22  Generally  benefical.  Benefit  in  proportion  to 

size  of  program. Good  management  practices  and 
good  source  of  management  information.  Using 
MPP  changed  undisciplined  process  to  produce 
reliable,  consistent  software.  Should  result 
in  lower  maintenance  costs. 

15  Neutral  or  no  comment. 

11  MPP  should  be  tailored  for  the  application.  Modes 

of  application  should  be  flexible.  A subset  of 
MPP  could  be  cost-effectively  used  in  unique 
applications. 

MPP  can  be  costly  to  establish  and  enforce.  Can 
increase  work  and  decrease  productivity.  Some 
MPP  emphasized  low  leverage  activities.  Some 
standards  inferior  to  normal  practices. 

Other  factors  more  important  than  MPP,  e.g., 
work  environment,  stability  of  software  re- 
quirements, professional  experience  and  manage- 
ment expertise. 

Disciplined  use  of  MPP  not  strongly  enforced. 

Lack  of  committment  to  achievement  of  long 
range  MPP  benefits.  MPP  application  neither  ex- 
tensive, nor  consistent  nor  early  enough  causing 
some  increased  cost  due  to  rework.  Independent 
testing  not  very  effective. 


11 


10 


8 


. sap,  t."/. ..  «*.  * w 


r 
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5 Need  for  better  documentation  and  training  on 

the  proper  application  of  MPP  to  achieve  more 
uniformity,  productivity  and  discipline. 

4 Requirements  changes  are  most  important  software 

development  problem  that  MPP  alone  can't  solve. 

4 Unit  Development  Folders  essential  for  required 

management  visibility  of  software  development 
status.  Process  Construction  Program  and  Code 
Auditor  necessary  to  ensure  code  compliance  with 
standards. 

2 Quality  Assurance  personnel  not  sufficiently 

qualified  (technically)  for  good  communication 
with  developers.  Configuration  Management  sys- 
tem too  slow  in  responding  to  change;  required 
fast-response  informal  change  control  system. 

1 Review  of  Unit  Development  Folders  (UDF)  re- 

quires extra  time  and  effort;  UDF  control  re- 
quires a large  bureaucracy. 

1 STP  afforded  excellent  opportunity  to  experiment 

(define,  apply,  evaluate,  improve)  on  a variety 
of  modern  practices,  thus  very  beneficial  for 
future  programs. 


t. 


From  the  above  it  can  be  seen  that  38  of  the  94  comments  (about 
40%)  were  Dositive  in  nature,  while  27  (about  29%)  were  negative  and  29 
(about  31%)  were  more  or  less  in  the  middle.  More  important  than  the  per- 
centages, however,  the  overall  message  in  the  comments  appears  to  confirm 
some  of  our  earlier  findings,  and  prompt  new  ones  namely: 

• The  proper  handling  (i.e.,  baselining  and  controlling  changes) 
of  software  requirements  is  of  critical  importance  with  respect 
to  software  quality,  cost  and  schedule,  and  MPP  which  encourage 
and  support  proper  handling  (such  as  Baselining  of  Requirements 
Specification  and  Requirements  Analysis  and  Validation)  are 
most  needed  and  have  most  beneficial  impact. 

• Although  there  is  substantial  agreement  on  the  overall  beneficial 
impact  of  the  11  STP  MPP,  there  is  clearly  room  for  improvement, 
especially  regarding  the  four  least  highly  ranked  MPP  (Independent 
Testing,  Enforced  Programming  Standards,  Software  Configuration 
Management,  and  Formal  Inspection  of  Documentation  and  Code). 

This  is  consistent  with  the  MPP  Impact  Evaluation  summary  which 
generally  indicates  higher  positive  MPP  impact  (in  the  theoreti- 
cal case)  than  that  reported  in  the  actual  STP  software  de- 
velopment to  date. 

• Most  of  the  positive  attitude  toward  and  general  acceptance  of 
MPP  is  expressed  in  terms  of  positive  impact  on  software  quality 
(particularly  reliability  and  maintainability),  while  it  is 
apparently  felt  that  MPP  contribute  less  directly  to  software 
development  cost  and  schedule.  Furthermore,  if  MPP  are  introduced 
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late  in  the  development  process,  or  if  they  are  poorly  defined  or 
poorly  understood  and  inconsistently  applied,  they  can  adversely 
effect  rework,  productivity,  and  personnel  motivation  with  pos- 
sible negative  impact  on  cost  and  schedule. 


3.3.3  Correlation  of  Study  Results 

The  preceding  sections  reported  upon  two  independent  but  similar 
impact  evaluation  activities.  The  first  provided  an  assessment  of  the  im- 
pact of  detailed  coding  and  documentation  standards  on  desirable  character- 
istics of  end  item  software  and  the  software  development  process.  The  second 
provided  an  assessment  of  the  impact  of  higher  level  modern  programming 
practices  on  avoidance  or  elimination  of  typical  problems  that  plague  soft- 
ware development.  The  two  activities  were  similar  in  that  they  sought  quan- 
titative data  to  permit  completion  (i.e.,  filling  in  the  blank)  of  statements 
of  the  general  form: 

"It  can  be  asserted  with confidence  that  practice  X has 

impact  on  software  characteristic  (or  problem)  Y." 

For  example,  from  the  Programming  Standards  Impact  Evaluation  we  obtained 
an  assertion  strength/influence  rating  of  Medium/Positive  as  the  perceived 
average  impact  of  programming  standards  on  the  software  characteri Stic 
"Code  Maintainability/Usability".  From  the  MPP  Impact  Evaluation  we  obtained 
two  impact  ratings  for  the  practice  "Enforced  Programming  Standards"  on  the 
typical  software  production  problem  "Unmaintainability".  These  were  Strong/ 

. Positive  (based  on  actual  STP  experience)  and  Strong/Strong  Positive 

’ (estimating  the  theoretical  impact  under  optimum  circumstances). 

There  were  enough  similarities  between  the  surveys  to  permit  a 
direct  comparison  and  simple  illustration  of  the  highly  correlated  results. 

< That  is,  it  is  possible  to  compare  the  average  impact  of  programming  stan- 

dards on  a software  characteristic  with  both  the  actual  and  theoretical 
impact  of  Enforced  Programming  Standards  on  a problem  strongly  related  to 
the  characteristic.  In  the  illustration  of  the  comparison  (Figure  3-12.  ), 
the  letter  "C"  indicates  the  composite  (i.e.,  average)  assertion  strength/ 
influence  rating  from  row  19  of  the  Programming  Standards  Impact  Evaluation 
matrix.  The  letters  "A"  and  "T"  indicate  the  ratings  from  corresponding 
column  elements  in  row  6 of  the  MPP  Impact  Evaluation  matrix.  The  y-axis 
contains  an  ordering  of  the  possible  impact  ratings  from  very  positive  to 
very  negative  with  all  Inconclusive  ratings  combined  as  a single  point  on 
the  scale,  and  the  x-axis  contains  seven  areas  of  commonality  between  the 
characteristics  and  problems  in  the  two  surveys. 

The  apparent  correlation  of  the  survey  results  is  very  good,  es- 
pecially in  view  of  several  important  differences  in  the  surveys; 

• The  "C"  ratings  represent  the  average  influence  of  all  18 
standards  on  each  characteristic,  wnile  the  "A"  and  "T"  ratings 
are  for  the  total  influence  of  Enforced  Programming  Standards 
on  each  problem. 

• The  first  survey  was  primarily  concerned  with  initial  development 
cost  and  schedule,  while  the  MPP  survey  called  explicitly  for 
impact  assessments  with  respect  to  life  cycle  cost  and  schedule 
concerns. 
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Figure  3-12.  Correlation  of  Survey  Results  for  "Enforced  Programming  Standards 
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4.0  MPP  COMPARISON  METHODOLOGY 


4 . 1 Objective  and  Approach 

One  of  the  very  important  aims  of  The  Impact  of  MPP  on  System 
Development  study  was  to  develop  a framework  within  which  various  practices 
and  overall  MPP  implementations  could  be  compared.  To  begin  to  solve  the 
problem  of  selecting  appropriate  practices  for  a particular  application, 
it  was  necessary  to  provide  an  MPP  comparison  methodology  that  would  permit: 

• detailed  comparative  analysis  of  two  or  more  individual  prac- 
tices, and 

• overall  comparison  of  the  combined  effectiveness  of  alternative 
MPP  implementations. 

It  was  expected  that  such  a methodology  would  represent  significant  progress 
toward  more  scientific  and  systematic  evaluation  and  selection  of  specific 
practices  which  address  the  range  of  problems  that  have  historically  plagued 
software  production.  Unfortunately,  in  the  past,  very  little  attention  has 
been  given  to  determining  the  actual  nature  of  the  effect  of  individual 
practices  on  distinct  software  production  problems.  What  has  been  missing  is 
an  appraisal  of  the  strengths  and  weaknesses  of  practices  and  the  use  of 
the  appraisal  to  guide  selection  of  appropriate  practices  for  application  to 
software  projects  of  varying  type,  size  and  complexity. 

Clearly,  all  modern  programming  practices  are  not  equally  effective 
in  all  respects.  It  is  recognized  that  different  people  have  different 
strengths  and  weaknesses,  and  the  same  is  true  for  modern  programming 
practices.  In  fact,  this  phenomenon  is  especially  evident  for  practices  since 
they  are  usually  invented  for  a special  reason  (e.g.,  to  enhance  some  parti- 
cular software  quality  or  to  solve  some  particular  production  problem)  with- 
out regard  for  other  possible  benefits  and  side  effects. 

The  approach  to  development  of  both  the  MPP  impact  evaluation  metho- 
dology and  the  MPP  comparison  methodology  was  strongly  influenced  by  the  above 
consideration.  The  approach  was  to  identify  software  production  areas,  within 
which  comparison  of  practices  could  be  accomplished  in  terms  of  individual 
strengths  and  weaknesses.  This  involved  the  development  of  a model  that  would 
serve  to  both  definitize  and  normalize  the  impact  of  practices  on  software 
development.  It  also  involved  the  definition  of  a procedure  for  using  the 
model  to  compare  MPP  implementations  to  aid  in  the  selection  of  appropriate 
practices  for  different  software  projects. 

4.2  The  Comparison  Model  and  Procedure 

The  basic  element  of  the  comparison  model  is  a generalized  version 
of  the  impact  evaluation  matrix.  Specific  examples  of  the  matrix  are  given 
elsewhere  in  this  report  in  presenting  the  results  of  the  Programming 
Standards  and  MPP  Impact  Evaluation  activities.  In  both  cases,  each  row  of 
the  matrix  corresponded  to  a practice,  and  each  column  corresponded  to  a 
characteristic  of  software  production(i .e. , a quality  of  developed  software, 
an  attribute  of  the  software  development  process,  or  a typical  problem 
plaguing  software  development).  The  generalized  version  of  the  matrix  for 
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comparative  analysis  of  MPP  implementations  can  be  viewed  as  a combination 
of  the  impact  evaluation  matrices  for  two  sets  of  MPP  as  illustrated  in 
Figure  4-1.  Each  element  of  the  matrix  contains  five  numbers  (i.e.,  a 
distribution  of  frequencies)  corresponding  to  the  influence  categories: 

• High  positive 

• Low  positive 

• No  contribution 

• Low  negative 

• High  negative 

The  procedure  for  performing  a comparative  analysis  of  the  two  MPP 
implementations  involves  computation  of  an  overall  figure  of  merit  for  each 
MPP  set.  In  order  to  permit  comparative  evaluation  of  selected  MPP  in  the 
context  of  differing  types  of  software  development  activities,  the  procedure 
also  involves  an  assignment  of  weights  to  each  column  (characteristic)  and 
each  row  (practice).  Each  column  weight  is  a number  between  1 and  99  which 
reflects  the  relative  importance  of  the  characteristic  in  a particular  soft- 
ware development  context.  Each  row  weight  is  either  1 or  0 to  indicate  whether 
or  not  the  influence  of  the  corresponding  practice  is  to  be  used  in  computing 
an  overall  figure  of  merit  for  the  MPP  set. 

Given  a set  of  column  weights  (CW.  for  i=l,....,N)  a gross  comparison 
of  the  two  MPP  implementations  can  be  obtained  from  two  applications  of  the 
hypothesis  test  performed  by  the  MERIT  support  program.  A figure  of  merit 
for  the  first  MPP  implementation  (FOM^)  is  obtained  by  setting  the  first 
< Mi  row  weights  equal  to  1 and  the  remaining  row  weights  equal  to  0.  Similarly, 

FOM2  is  obtained  by  setting  RW-j =0  for  i=l,....,Mi  and  the  remaining  row 
weights  equal  to  1.  In  addition  to  the  overall  FOM,  MERIT  produces  a com- 
posite FOM  for  each  row  (i.e.,  the  summary  influence  of  each  practice  on  all 
characteristics)  and  for  each  column  (i.e.,  the  summary  influence  of  all 
practices  on  each  characteristic).  Implicit  in  the  calculation  of  the  figures 
of  merit  is  the  basic  assumption  that  the  cumulative  impact  of  several  prac- 
tices is  strictly  additive  (i.e.,  not  mutually  synergistic  or  overlapping). 

A comparative  analysis  of  the  two  MPP  implementations  is  accomplished  through 
review  of  the  two  sets  of  MERIT  output  and  direct  comparison  of  the  overall 
figures  of  merit  as  well  as  those  for  each  column.  In  addition,  comparison 
of  figures  of  merit  for  the  rows  provides  an  indication  of  the  relative 
influence  of  alternative  practices  on  all  characteristics. 
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4.3  An  Example  of  the  Comparison  Methodology 

To  illustrate  the  comparison  methodology,  the  set  of  11  STP  modern 
programming  practices  was  arbitrarily  divided  into  two  MPP  implementations 
with  MPP  Set  #1  including: 


• 

Requirements  Analysis  and  Validation 

- P11 

• 

Process  Design 

" P12 

• 

Incremental  Development 

" P13 

• 

Complete  Preliminary  Design 

- P14 

• 

Unit  Development  Folders 

- P15 

and  MPP  Set 

#2  including: 

• 

Enforced  Programming  Standards 

' P21 

t 

Independent  Testing 

- P22 

• 

Software  Configuration  Management 

‘ P23 

• 

Baselining  of  Requirements  Specifications 

- p24 

0 

Formal  Inspection  of  Documentation  and  Code 

' P25 

0 

Software  Development  Tools 

" P26 

In  addition,  to  illustrate  the  potential  for  analysis  of  the 
potential  impact  of  MPP  in  the  context  of  differing  types  of  projects, 
new  weights  were  assigned  to  the  characteristics  (i.e.,  the  typical  problems 
plaguing  software  development)  to  reflect  the  relative  significance  of  the 
problems  in  a hypothetical  development  activity.  The  weights  used  for  this 
example  were  arbitrarily  chosen  as  the  following: 

Characteristic  (Problem) 

Weiqht 

Cl 

(Cost  Overrun) 

§9 

C2 

(Development  Status  Invisibility) 

50 

C3 

(Unrel iabi 1 i ty) 

75 

C4 

(Unmaintainability) 

25 

C5 

(Inadequate  Satisfaction  of  Real  Requirements) 

50 

C6 

(Inefficient  Use  of  Resources) 

75 

C7 

(Schedule  Overrun) 

75 

i»,yrT 
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Cg  (Inadequate  Planninq  and  Control)  1 

Cg  (Project  Mismanagement)  1 

C^q  (Lack  of  Programming  Discipline)  1 

(Lack  of  Conclusive  Testing)  25 

Cj2  (Poor  Documentation)  25 


The  MERIT  program  was  used  to  apply  the  appropriate  row  and  column 
weights  to  the  individual  influence  distributions,  compute  composite 
distributions  for  each  row  and  column  and  for  the  entire  matrix,  and  per- 
form the  hypothesis  test  distributional  algorithm  in  deriving  for  each  set 
of  MPP: 


• an  FOM  (assertion  strength/influence  rating)  for  each  practice 
on  the  weighted  problems, 

• an  FOM  for  all  practices  included  in  the  particular  MPP  set  on 
each  problem,  and 


• an  overall  FOM  for  the  combination  of  all  pratices  included  in 
the  particular  MPP  set  on  the  combined  set  of  weighted  problems. 

The  key  analysis  results  are  shown  below,  and  complete  MERIT  output  is  in 
Appendix  E. 

Some  striking  differences  are  seen  in  comparison  of  the  asserted 
impact  of  the  combined  practices  of  Set  1 versus  the  combined  practices  of 
Set  2 and  in  comparison  of  the  overall  figures  of  merit  for  the  two  MPP 
implementations.  The  summary  figures  of  merit  (i.e.,  assertion  strength/ 
influence  rating)  obtained  from  the  hypothesis  test  performed  by  MERIT  were 
as  follows: 


Characteristic  (Problem) 

(Cost  Overrun) 

C^  (Development  Status  Invisibility) 

Cg  (Unrel iabi 1 i ty) 

C^  (Unmaintainability) 

Cg  (Inadequate  Satisfaction  of 
Real  Requirements) 

Cg  (Inefficient  Use  of  Resources) 

C (Schedule  Overrun) 


MPP  Set  #1  MPP  Set  *2 

Strong/Positive  Stpong/Positive 

Strong/Strong  Positive  Medium/positive 

Strong/Positive  Strong/Positive 

Med i urn/ Positive  Med i urn/ Positive 

Strong/Pos i ti ve  Medi um/Posi ti ve 


Strong/Positive  Medium/Positive 

Strong/Positive  Med i urn/ Positive 
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g (Inadequate  Planning  and  Control) 

Strong/  Positive 

Medium/Positive 

g (Project  Mismanagement) 

Medium/Positive 

Medium/Positive 

jg  (Lack  of  Programming  Discipline) 

Medium/Positive 

Medium/Positive 

, ^ (Lack  of  Conclusive  Testing) 

Strong/Positive 

Medium/Positive 

^2  (Poor  Documentation) 

Strong/Positive 

Medi um/Posi ti ve 

The  most  noticeable  difference  is  between  the  assertion  strength/ 
influence  rating  for  the  MPP  Sets  on  Characteristic  2 (Development  Status 
Invisibility)  which  prompt  the  following: 


• We  have  strong  confidence  that  the  5 practices  of  MPP  Set  1, 
in  combination,  make  a strong  positive  contribution  toward 
eliminating  the  problem  of  Development  Status  Invisibility. 


• We  have  med i urn  confidence  that  the  6 practices  of  MPP  Set  2,  in 
combination,  make  a positive  contribution  toward  eliminating  the 
problem  of  Development  Status  Invisibility. 


The  result  of  weighting  the  individual  influence  distributions  in 
computing  summary  FOMs  for  each  practice  and  a composite  FOM  for  the  two 
MPP  sets  is  shown  below:  , 

Weighted  Figure 

Practice"  of  Merit 

Pjj  - Requirements  Analysis  and  Validation  Strong/Positive 

?12  - Process  Design  Strong/Positive 

Pj2  - Incremental  Development  Strong/Positive 

Pj4  - Complete  Preliminary  Design  Strong/Positive 


P15  ‘ llit  Development  Folders 


Strong/Positive 


Composite  (MPP  Set  1)  Strong/Positive 

P21  " Enforced  Programming  Standards  Medium/Positive 

P22  - Independent  Testing  Medium/Positive 

P23  - Software  Configuration  Management  Medium/Positive 

P24  - Baselining  of  Requirements  Specifications  Strong/Strong  Pos. 


Weighted  Figure 

Practice  of  Merit 


P25  ~ fr°rma^  Inspection  of  Documentation  & Code  Medium/Positive 
P^g  - Software  Development  Tools  Strong  Positive 


] Composite  (MPP  Set  2) Medium/Positive 

The  composite  figures  of  merit  for  the  two  MPP  implementations  are 
indeed  different,  and,  in  the  context  of  a hypothetical  software  develop- 
ment activity  (for  which  the  weights  are  relevant): 

• We  have  strong  confidence  that  the  5 practices  of  MPP  Set  1,  in 
combination,  make  a positive  contribution  toward  eliminating 
the  problems. 

• We  have  med i urn  confidence  that  the  6 practices  of  MPP  Set  2,  in 
combination,  make  a positive  contribution  toward  eliminating  the 
problems. 

4.4  Limitations  of  the  Comparison  Methodology 

It  was  indicated  earlier  that  the  MPP  comparison  model  could  be 
used  for  the  selection  of  a set  of  appropriate  practices  for  a proposed 
software  development  activity.  This  involves  computation  of  an  overall  FOM 
for  candidate  combinations  of  MPP  and  selection  of  the  combination  with 
the  best  FOM.  It  is  important  to  note,  however,  that  the  validity  of  the 
FOM  computation  is  dependent  upon  the  make  up  of  the  individual  matrix 
elements  (i.e.,  the  influence  rating  frequency  distributions)  which  are 
sunned  to  form  an  overall  distribution.  In  particular,  the  sources  of  the 
impact  evaluation  ratings  may  be  different  and,  unless  the  frequency  distri- 
butions are  normalized,  the  influence  of  one  practice  may  unduly  dominate 
the  composite  FOM.  This  problem  can  be  avoided  by  appropriately  factoring 
the  individual  frequency  distributions  as  necessary  to  cause  the  sum  of  the 
frequencies  in  each  matrix  element  to  ’'e  a constant.  For  example,  the  three 
unnormalized  distributions  below  can  be  normalized  as  shown: 

Unnormalized  Frequency 


Distributions 

Composite 

% 

High  Positive 

14 

4 

1 

15 

15.2 

Low  Positive 

26 

8 

4 

38 

30.4 

No  Contribution 

22 

20 

14 

56 

44.8 

Low  Negative 

7 

3 

1 

11 

8.8 

High  Negative 

1 

0 

0 

1 

.8 

Total 

70 

35 

20 

125 

.n.. 
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Normalized  Frequency 
Distributions 


High  Positive 

14 

8 

3.5 

Low  Positive 

26 

16 

14 

No  Contribution 

22 

40 

49 

Low  Negative 

7 

6 

3.5 

High  Negative 

1 

0 

, 0 

Total 

70 

70 

70 

Composite 

25.5 

56 

111 

16.5 
1 

210.0 


12.1 

26.7 

52.9 

7.9 

.4 


The  importance  of  normalizing  the  contents  of  the  impact  evaluation  matrix 
is  more  apparent  f»om  comparison  of  the  differing  percentages  for  the  un- 
normalized and  normalized  composite  frequency  distributions  in  the  above 
example. 

An  alternative  and  easier  way  to  avoid  adding  "apples  and  oranges" 
is  to  use  the  percentage  figures  computed  for  each  frequency  distribution 
in  formulating  the  composite  FOM,  divide  by  the  number  of  matrix  elements 
in  the  sum  (i.e.,  compute  average  percentages),  and  apply  the  hypothesis 
test  to  the  percentages  rather  than  to  the  summed  frequencies.  The  MERIT 
program  does  not  currently  provide  this  capability  but  could  be  easily 
■(  modified  to  do  so. 

A secondary  limitation  on  the  utility  of  the  MPP  comparison  metho- 
dology arises  from  the  strict  dependence  on  the  completed  impact  evaluation 
matrix.  There  are,  of  course,  alternative  methods  for  acquiring  impact 
evaluation  data  including  the  modified  Delphi/survey/selective  validation 
approach  used  in  this  study,  experimentation  and  data  collection  on  parallel 
development  activities,  and  detailed  analysis  of  a variety  of  software 
developments  using  various  MPP  combinations.  All  of  these  methods  are 
aimed  at  isolating  and  measuring  the  effect  of  individual  practices  on  soft- 
ware production,  and  all  are  difficult  and  time  consuminq  to  apply. 

Finally  it  should  be  noted  that  the  two  sets  of  characteristics 
used  in  this  study  (i.e.,  the  desirable  qualities  of  software  and  the 
development  process  and  the  typical  problems  plaguing  software  development) 
are  certainly  not  exhaustive.  Although  the  12  typical  problems  served  well 
as  metrics  for  evaluating  the  impact  of  selected  STP  modern  programming 
practices,  it  is  possible  that  the  greatest  strength  (or  weakness)  of  some 
practice  cannot  be  readily  represented  without  expanding  the  list  of  char- 
acteristics. Thus,  if  many  comparisons  of  MPP  implementations  are  antici- 
pated, it  is  necessary  to  first  establish  a comprehensive  set  of  common 
characteristics  upon  which  the  detailed  assessment  of  a wide  variety  of 
practices  can  be  based  as  necessary  to  permit  a fair  comparison  of  re- 
lative impact. 
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5.0  CONCLUSIONS  AND  RECOMMENDATIONS 


5.1  Conclusions 

Of  the  more  than  30  candidate  modern  programming  practices  consid- 
ered during  this  study,  the  following  four  were  determined  to  have  the 
strongest  positive  impact  on  software  quality  and  productivity: 

1.  Requirements  Analysis  and  Validation 

2.  Baselining  of  Requirements  Specification 

3.  Complete  Preliminary  Design 

4.  Process  Design 

This  outcome  is  consistent  with  data  available  from  other  projects  [5,17] 
indicating  that  increased  rigor  and  engineering  discipline  in  the  re- 
quirements definition  and  design  phases  have  the  highest  leverage  on 
improving  software  production. 

The  following  three  MPP  were  determined  to  have  the  next  highest 

impact: 

5.  Incremental  Development 

6.  Unit  Development  Folders 

7.  Software  Development  Tools 

The  following  four  MPP  completed  the  list  of  the  highest-impact 
practices: 

8.  Independent  Testing 

9.  Enforced  Programming  Standards 

10.  Software  Configuration  Management 

11.  Formal  Inspection  of  Documentation  and  Code 

All  eleven  of  these  MPP  were  determined  to  have  a generally  positive 
impact  on  attaining  desirable  qualities  of  software  and  achieving  more 
productive  developer  effort  through  elimination  of  typical  software  produc- 
tion problems.  A special  survey  and  analysis  of  the  impact  of  18  detailed 
programming  standards  on  30  specific  software  qualities  showed  a generally 
positive  impact  on  software  quality  and  a mixed  impact  on  software  develop- 
ment productivity.  Noting,  however,  that  improved  software  quality  strongly 
correlates  with  long  range  benefits  such  as  easier  software  maintenance, 
r.  the  net  result  is  an  overall  positive  impact  of  the  standards  on  software 

life-cycle  productivity. 
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The  detailed  results  provide  hundreds  of  impact  assessments  for 
individual  MPP  that  are  difficult  to  abstract  here,  but  which  provide 
detailed  support  to  decision  makers  concerned  with  selection,  definition, 
and  implementation  of  appropriate  standards  and  procedures  for  individual 
projects.  Clearly,  the  assessment  of  a large  number  of  individual  inter- 
actions in  the  analysis  of  candidate  combinations  of  MPP  can  be  impractical 
to  handle  by  manual  methods.  This  problem  was  addressed  in  the  study  by 
developing  several  easy-to-use  computer  programs  to  support  cumulative  MPP 
impact  assessments  and  comparative  evaluation  of  MPP  implementations. 

5.2  Discussion  of  Conclusions 

It  should  not  be  surprising  that  the  overall  evaluation  of  the  STP 
MPP  turned  out  to  be  positive,  mainly  because  STP  was  never  intended  to  be 
a trial  experiment  with  a set  of  practices  having  unknown  effect.  Indeed, 
a major  emphasis  of  the  project  was  to  evolve  an  improving  set  of  practices 
to  eliminate  a wide  range  of  software  production  problems,  to  enhance  speci- 
fic software  qualities  and  to  reduce  the  life  cycle  cost  of  the  software. 

This  evolutionary  approach  has  had  several  effects.  The  first  and 
most  obvious  is  that  an  evaluation  of  any  set  of  relatively  mature  practices 
will  almost  certainly  produce  a positive  impact  rating,  because,  in  general, 
the  practices  are  designed,  applied,  examined,  and  refined  with  that  goal 
in  mind.  Another  more  subtle  but  important  observation  concerns  the  evaluated 
impact  of  individual  practices. 

Before  a newly  implemented  practice  becomes  mature  (i.e.,  during 
its  initial  application),  its  value  is  strongly  governed  by  two  competing 
phenomena:  the  Hawthorne  effect  [18],  and  a resistance  to  change/learning  curve 
effect.  The  first  results  in  emphasizing  (often  overemphasizing)  the  strengths 
of  the  practice  while  resistance  to  change  has  generally  the  opposite  effect. 
This  competition  is  healthy  since  it  leads  to  steady  improvement  of  the 
practice  through  refinements  which  maximize  the  strengths  and  minimize  the 
weaknesses.  The  evolutionary  process  involves  many  people,  however  and  it 
is  sometimes  painful  and  usually  slow.  Thus,  the  premature  evaluation  of  a 
practice,  if  objective,  will  often  be  mixed,  having  positive  impact  in  some 
areas,  negative  impact  in  others.  In  some  areas  the  impact  may  be  inconclu- 
sive (i.e.,  neither  clearly  positive  nor  negative)  due  to  differing  exper- 
iences and  insufficient  data. 

The  best  example  of  the  latter  phenomenon  occurred  in  the  evolution 
of  the  structured  coding  requirement.  It  was  the  most  recently  adopted  STP 
standard  programming  practice.  The  evaluated  impact  of  the  structured  coding 
practice  was  (as  shown  in  Section  3)  either  negative  or  inconclusive  with 
respect  to  one  third  of  the  desirable  qualities  of  software  and  the  develop- 
ment process.  Among  the  factors  which  also  influenced  the  structured  pro- 
gramming requirement  were: 

• The  formal  requirement  for  structured  coding  was  established 
two  years  after  the  start  of  the  project,  after  a large  amount 
of  code  had  been  developed. 


• The  structuring  standard  was  imposed  on  coding  in  either 
FORTRAN  or  assembly  language,  nei thereof  which  easily  support 
writing  structured  programs,  and  / 

• The  phase  of  the  project  during  which  the  major  benefits  of 
structured  programming  are  expected  to  be  reaped  (i.e.,  during 
operations  and  maintenance)  has  not  yet  begun. 

Another  important  feature  of  the  study  results  is  the  grouping  of 
the  eleven  MPP  into  three  broad  categories.  The  four  highest  ranked  prac- 
tices (Requirements  Analysis  and  Validation,  Baselining  of  Requirements 
Specifications,  Complete  Preliminary  Design,  and  Process  Design)  fell  into 
one  group;  the  three  middle  ranked  practices  (Incremental  Development,  Unit 
Development  Folders,  and  Software  Development  Tools)  in  another;  and  the 
four  lowest  ranked  practices  (Formal  Inspection  of  Documentation  and  Code, 
Software  Configuration  Management,  Independent  Testing,  and  Enforced  Pro- 
gramming Standards)  fell  in  the  bottom  group.  The  significant  point  to  be 
noted  is  that  the  highest  ranked  group  contains  practices  that  are  generally 
applied  at  the  front  end  of  the  software  development  activity,  the  middle 
ranked  group  applies  to  the  middle  stages  of  development,  while  the  lowest 
ranked  group  applies  mostly  to  the  latter  stages. 

Notice  also  that  the  first  group,  and,  to  some  extent,  the  second 
group  of  practices  are  viewed  as  providing  assistance  to  the  human  perform- 
ance of  tasks  generally  thought  of  as  complex,  thought-provoking  and  highly 
creative.  The  last  group,  on  the  other  hand  is  more  often  viewed  as  the 
implementation  of  policing  functions  aimed  at  checking  up  on  developers  and 
pointing  out  their  mistakes.  It  is  likely  that  a practice-popularity-poll 
would  correlate  well  with  the  actual  ranking  of  the  groups  according  to 
their  relative  positive  impact  on  software  development. 

The  relative  impact  ranking  of  the  groups  of  practices  was  also 
strongly  influenced  by  their  history  of  application  in  the  STP  chronology 
(Section  2).  In  particular,  the  software  development  activity  has  been 
influenced  by  the  first  group  of  practices  for  the  longest  period  of  time, 
and  those  practices,  therefore,  have  had  more  opportunity  to  make  their 
positive  impact  felt.  This  observation  is  consistent  with  recent  software 
engineering  research  findings  (e.g.,  CC I P-85  and  the  recently  completed 
TRW  Software  Reliability  Study  for  RADC)  which  point  out  1)  the  large 
fraction  (as  high  as  70%)  of  total  software  problems  which  occur  early  in 
the  development  cycle,  and  2)  the  unusually  high  value  of  practices  that 
help  find  problems  as  early  as  possible,  before  they  propagate  confusion, 
additional  work,  and  expense  throughout  subsequent  project  phases  [17 1 19] # 

Another  important  conclusion  of  the  study  is  that  the  isolation  of 
the  effect  and  evaluation  of  impact  of  any  given  practice  within  a large, 
complex  and  dynamic  project  environment  can  be  exceedingly  difficult. 
Fortunately  this  conclusion  was  anticipated  at  the  outset  of  the  study,  and 
great  care  was  taken  to  identify  a select  subset  of  the  many  STP  practices 
such  that: 


t£a  , : 


PAGE  5 


4 


• The  number  of  practices  was  small  enough  to  permit  thorough 
study  of  each, 

• Each  practice  was  generic  enough  (i.e.,  not  strongly  oriented 
toward  unique  features  of  STP)  to  permit  relatively  straight- 
forward inclusion  in  Air  Force  development  guidelines  with 
broad  applicability, 

• Each  practice  had  been  used  extensively  enough  to  ensure  a 
reasonable  base  of  project  data  and  experience  to  support  impact 
evaluation,  and 

t The  set  of  practices,  in  combined  application,  spanned  the 
development  process  and  addressed  each  major  development 
phase. 

The  careful  selection  of  MPP  limited  the  number  of  practices  a lot  and 
limited  their  scope  a little,  but  it  permitted  sufficient  depth  of  the 
impact  analyses  and  sufficient  effort  for  comparison  methodology  devel- 
opment to  make  it  possible  to  ultimately  conclude: 

• The  STP  MPP  (and  some  practices  in  particular)  contribute 
significantly  to  enhancement  of  important  qualities  of  soft- 
ware, and  generally  the  investment  required  to  implement  and 
apply  MPP  pays  off  in  overall  life-cycle  cost  reductions. 

• There  are  significant  differences  among  MPP  in  both  the  areas 
to  which  they  contribute  and  the  magnitude  of  the  contributions 
they  make. 

t It  has  been  demonstrated  that  a methodology  can  be  developed 
and  used  to  obtain  a comparative  evaluation  of  alternative 
MPP  implementations. 

• The  MPP  comparison  methodology  (including  the  support  tools 
developed  for  this  study)  can  be  used  to  both  select  and 
evaluate  candidate  combinations  of  MPP  for  projects  of 
varying  type,  size  and  complexity. 


5.3  Relation  of  Conclusions  to  Current  Air  Force  Practices 


Some  of  the  MPP  determined  in  this  study  to  have  high  positive 
impact  are  already  fairly  well  covered  by  established  Air  Force  regulations 
such  as  MS-483  and  MS-490.  These  are: 

#2.  Baselining  of  Requirements  Specification 

#10.  Software  Configuration  Management 


Other  of  the  MPP  have  been  covered  in  some  of  the  more  recent  Air 
Force  and  DoD  initiatives  such  as  AFR  800.14,  DoDD  5000.29,  MS-52779,  and 
the  recent  RADC  standards  effort.  These  are: 
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#9.  Enforced  Programming  Standards 

#11.  Formal  Inspection  of  Documentation  and  Code 

Some  of  the  MPP  are  sufficiently  wel 1 -formal i zed  and  general-purpose 
to  be  covered  in  new  Air  Force  policy  initiatives.  These  are: 

#5.  Incremental  Development 

#6.  Unit  Development  Folders 

#7.  Software  Development  Tools  (partly  — e.g.,  test  tools,  code 
auditing  tools) 

Still  other  of  the  MPP  need  and  deserve  further  formalization  and/or  inves- 
tigation of  their  general  applicability  prior  to  being  established  in  policy 
initiatives.  These  are: 

#4.  Process  Design 

#7.  Software  Development  Tools  (partly — e.g.,  highly  specialized 
simulators  and  post  processors) 

5.4  Recommendations 

The  overall  objective  of  the  MPP  study  was  to  assess  the  effects 
of  MPP  on  a large  DoD  system  software  development  so  that  those  practices 
determined  to  have  a favorable  effect  could  be  recommended  to  and  considered 
by  the  Air  Force  for  adoption  on  future  developments.  In  light  of  this  goal 
and  owing  to  the  determination  of  positive  impact  for  all  eleven  of  the  STP 
MPP,  it  is  reasonable  to  recommend  that  all  be  considered  for  inclusion  in 
Air  Force  software  development  guidelines,  procurement  practices,  and  policy 
initiatives.  Of  these  MPP,  the  four  most  highly  ranked  should  receive  es- 
pecially strong  consideration  in  view  of  the  pressing  need  for  improved 
methods  for  defining,  expressing  and  allocating  software  requirements  and 
for  maintaining  their  traceability  throughout  the  development  process.  On 
the  other  hand,  it  is  clear  that  not  all  software  development  activities 
are  alike,  and  few  systems  compare  with  the  size,  complexity  and  real  time 
performance  requirements  of  the  software  being  developed  by  TRW  on  the  Sys- 
tems Technology  Program. 

Based  in  part  on  the  above  consideration  as  well  as  on  the  detailed 
MPP  study  findings  documented  fully  in  this  report,  it  is  recommended  that 
the  Air  Force: 

1.  Continue  to  emphasize  those  high  impact  MPP  listed  above  which 
are  now  covered  or  will  be  covered  in  existing  and  new  Air  Force 
and  DoD  policies 

2.  Emphasize  the  following  additional  high  impact  MPP  in  future  Air 
Force  policy  initiatives: 

• Incremental  Development 

• Unit  Development  Folders 

• Software  Development  Tools  (especially  those  with  demon- 
strated capability  and  broad  applicability) 
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3.  Promote  research  to  formalize  and  generalize  other  high  impact 
MPP: 

• Process  Design 

• Software  Development  Tools  (especially  requirements 
definition  and  design  aids,  higher  order  languages  and 
diagnostic  language  processors,  and  advanced  test  tools) 

4.  Support  continued  investigation  necessary  for  further  confirma- 
tion of  the  results  of  this  study  and  comparison  with  other 
results  on  other  projects 

5.  Develop  standardized  definitions  and  measures  (e.g.,  productivity, 
modularity,  cost)  critical  to  accurate  and  comparable  assessments 
of  impact  of  alternative  MPP 

6.  Pursue  evaluation  of  MPP  used  by  other  contractors  on  other  projects 
and  build  upon  the  impact  evaluation/comparison  framework  using 

the  tools  and  data  base  developed  in  this  study 

7.  Establish  ground  rules  and  a methodology  for  relating  software 
qualities  to  software  operations  and  maintenance  costs  to  sup- 

. port  better  assessment  of  life-cycle  MPP  impact 

\ 8.  Initiate  research  and  development  of  techniques  for  selecting 

an  appropriate  subset  of  high  impact  practices  commensurate  with 
the  type,  size,  complexity,  and  available  resources  of  planned 
software  procurements. 
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Test  Support  Tools:  A comprehensi ve  set  of  support  tools  which  1)  help 
with  code  inspection  for  compliance  with  programming  standards  (Code 
Auditor,  Product  Assurance  Evaluator  (PACE)  STRUCT,  Process  Construction 
Program  (PCP)),  2)  provide  automated  assistance  in  preparation,  handling, 
execution  and  analysis  of  test  cases  and  test  results  including  automated 
retest  results  comparison  and  test  sufficiency  measurement  (PACE,  Logic 
Reduction  Analyzer,  Automated  Test  Case  Comparator  (ATCC),  Data  Reduction 
and  Report  Generation  (DRGG)),  and  3)  permit  simulation  and  detailed 
analysis  of  expected  performance  characteristics  and  resource  requirements 
for  candidate  design  solutions  (Data  Processing  Subsystem  Simulator  (DPSS) 
and  Analytical  Load  Formulation  (ALF)). 

Computer  Program  Assembly  List:  A management  planning  technique  and  auto- 
mated tool  to  ensure  early  identification  and  detailed  definition  of  total 
software  system  structure  complete  with  execution  time  and  core  storage 
allocations  and  responsible  personnel  for  each  program  element;  use  of  the 
CPAL  for  periodic  status  reporting  of  actuel  versus  estimated  computer 
resources  and  percent  complete  at  the  unit  level  and  above. 

Independent  Test:  A formal  organization  within  the  project  (but  separate 
from  design  and  development  organizations)  with  the  responsibility  for 
planning  and  performing  an  independent  test  and  evaluation  of  software 
for  satisfaction  of  requirements  as  well  as  responsibility  for  direction 
of  software  system  integration. 

Requirements  Traceability:  Use  of  documentation  techniques  and  functional 
processing  thread  diagrams  to  establish  and  maintain  forward  and  reverse 
traceability  from  requirements  through  design  to  code  and  from  require- 
ments to  validation  tests. 

Functional  Capabilities  List:  A technique  used  to  provide  objective  assur- 
ance of  sufficient  scope  and  depth  of  unit  testing  done  by  original  devel- 
opers. Identifies  functions  recognized  in  detailed  review  of  design 
documentation  and  specifies  tests  to  assure  that  corresponding  capabilities 
are  present  in  as-built  code. 

Tiger  Teams:  Special  small  teams  of  highly  skilled  system  engineers  and 
designers  organized  to  focus  needed  early  attention  on  critical  algorithms 
and  reduce  the  likelihood  of  serious  problems  later  in  the  development 
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process.  Typically  concentrate  on  validation  of  requirements  and  pre- 
liminary design  approach.  Design  and  code  candidate  algorithms  to  support 
evaluation  of  performance  characteristics. 

Incremental  Development:  Overall  software  production  approach  involving 
the  definition  of  progressively  more  complete  increments  (i.e.,  "loops") 
of  system  capability.  Each  loop  provides  an  independently  testable  in- 
crement of  the  evolving  program  containing  prototype  code  (stubs)  in 
place  of  functions  to  be  fully  developed  in  a subsequent  increment. 

Builds  within  loops,  carries  the  incremental  development  approach  to  more 
detail  and  further  reduces  the  chance  of  downstream  problems. 

A. 1.2  Programming  Standards 

Text  Format:  The  requirement  to  organize  and  document  descriptive  infor- 
mation on  software  modules  (subroutines,  routine,  tasks,  and  programs) 
in  accordance  with  the  prescribed  format  of  MIL-STD  490,  Specification 
Practices. 

I ' Text  Level  of  Detail : The  requirement  to  provide  descriptive  information 

l on  software  modules  to  fully  and  clearly  illustrate  their  purpose,  func- 

tion and  interfaces  with  other  modules  in  accordance  with  the  prescribed 

| 1 content  of  MIL-STD  490. 

Flow  Chart  Format:  The  requirement  to  develop  diagrams  depicting  the 
functional  hierarchy  and  execution  flow  of  all  software  modules  using 
the  flow  charting  symbols  prescribed  in  "The  International  Organization 
for  Standardization  Draft  Recommendation  on  Flowchart  Symbols  and  Their 
Usage  in  Information  Processing  (ANSI  X-3.5.1970)". 

Flow  Chart  Level  of  Detail:  The  requirement  for  1)  program  and  task  level 
flow  charts  to  depict  overall  composition  and  hierarchical  structure  down 
to  the  lowest  routine  level  with  a single  block  for  each  routine  con- 
taining the  routine  name  and  a brief  functional  description,  and  2)  routine 
level  flow  charts  to  illustrate  in  detail  the  basic  function  of  the  routine 
including  identification  of  all  external  files  used,  operating  system  ser- 
vice requests,  decision  logic  and  branches,  calculations,  calls  to  sub- 
routines, and  entry  and  exit  points. 
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Statement  Label  Format:  All  statement  labels  to  be  right  justified  in 
Columns  2-5  and  all  labels  to  appear  in  ascending  order  throughout  the 
extent  of  the  module. 

Executable  Statement  Format:  All  executable  FORTRAN  statements  to  start 


in  Column  7 except  where  indentation  is  required  to  illustrate  code  seg- 
ments and  nested  structures. 


Routine  Size  (Modularity):  Each  routine  to  contain  a maximum  of  100  ex- 
ecutable source  statements  from  the  time  it  is  initially  coded  until 
turnover  for  independent  testing  with  the  limit  subsequently  raised  to 
150  to  permit  incremental  addition  of  new  capability  without  major  soft- 
ware breakage. 


Calling  Sequence  Arguments:  Arguments  in  calling  statements  to  subroutines 
and  function  subprograms  not  to  contain  arithmetic  or  logical  expressions, 
and  routines  in  the  real-time  software  to  have  no  calling  sequence  arguments. 

^ Mixed  Mode  Arithmetic:  Mixed  mode  arithmetic  expressions  (excluding  integer 

exponentiation)  not  allowed  on  the  right  side  of  an  equal  sign. 

D0-LQ0P  Usage:  Do-Loops  not  to  exceed  six  levels  of  nesting,  and  Do-Loop 
index  parameters  to  be  expressed  only  as  integer  constants  or  variables. 

Computed  G0-T0  Usage:  All  computed  GO  TO  statements  to  be  immediately  pre- 
ceded by  a validity  (range)  test  of  the  variable  switch  parameter.  Tests 
revealing  invalid  parameter  values  to  initiate  error  processing.  Statement 
labels  used  as  branch  addresses  to  appear  after  (i.e.,  below)  the  GO  TO 
statement  in  the  sequence  in  which  they  appear  in  the  GO  TO  statement 
parameter  list,  with  the  exception  of  redundant  statement  labels  in  the 
parameter  list. 


use  of  blank  COMMON  not  permitted. 

Imbedded  Physical  Constants  Usage:  Literal  physical  constants  (e.g.,  3.14159 
for  the  value  of  pi)  not  to  be  imbedded  in  the  code. 


Preface  Commentary  Block  Requirement:  Each  module  to  contain  a standardized 
block  of  comment  statements  immediately  following  the  module  declaration 
statement.  The  block  to  contain:  1)  module  name  and  version  identifier, 


sSi 


s*;«  <•*-*"«»  - 


PAGE  A 


5 


2)  functional  description,  3)  inputs,  4)  outputs,  5)  key  local  variables, 

6)  usage  restrictions , 7)  identification  of  modules  called  by  this  one, 
and  8)  description  of  abnormal  return  conditions  and  actions. 

In-Line  Commentary  Requirement;  Each  module  to  contain  sufficient  in-line 
commentary  in  the  source  code  to  identify  the  purpose  of  every  statement 
which  conditionally  alters  a data  value  or  which  alters  the  sequential 
execution  of  program  statements. 

Structured  Coding  Requirement:  Each  module  to  be  structured  in  accordance 

with  a prescribed,  restricted  list  of  segment  structures  including:  1) 
sequence  of  two  operations,  2)condi tional  branch  and  rejoin,  3)  loop 
construct  with  condition  tested  prior  to  execution  of  loop  body,  4)  loop 
construct  with  condition  tested  after  execution  of  loop  body,  and  5)  loop 
construct  allowing  for  premature  escape  from  unnecessary  loop  repetition. 
Permits  nesting  of  segment  structures  to  any  level  and  error-condition 
transfers  from  any  segment  structure  to  a common  point  in  the  module. 

Execution  of  Every  Program  Branch  Requirement:  The  requirement  to  cause 
the  execution  of  every  program  branch  statement  and  to  force  the  transfer 
of  control  to  each  of  the  possible  branch  addresses  during  unit  testing 
of  each  module. 

Naming  Conventions:  The  requirement  to  formulate  the  names  of  processes, 
programs,  tasks,  routines,  real-time  events,  and  global  data  files  and 
parameters  in  accordance  with  established  patterns  prescribing  the  use  of 
alphanumeric  characters  identifying  item  type  and  function. 

A. 1.3  Detailed  Definitions  of  Final  11  MPP 

Requirements  Analysis  and  Validation:  The  formal  decomposition  and  alloca- 
tion of  system  requirements  into  structured  software  requirements  and  use 
of  automated  techniques  to  analyze  and  validate  the  requirements.  Software 
requirements  engineering  is  the  discipline  for  developing  a complete,  consis- 
tent, unambiguous  specification  which  can  serve  as  a requirements  baseline 
for  common  agreement  among  all  parties  concerned,  and  describing  "what" 
the  software  product  will  do.  Requirements  analysis  and  validation  is  the 
practice  through  which  a given  software  requirements  specification  can  be 
analyzed  to  check  for  completeness  and  consistency  and  validate  its  content 
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prior  to  design  and  coding.  It  is  a general  concensus  that  it  is  extremely 
cost  effective  to  analyze  thoroughly  the  given  requirements  and  detect 
errors  early  in  the  software  development  cycle.  The  lack  of  a good  re- 
requirements specification  makes  a top  down  design  impossible  and  testing 
meaningless,  obscures  management  monitoring  capability  and  prevents  good 
communication  between  user  and  developer. 

The  first  step  in  the  software  deve1  oment  process  is  the  definition  of 
data  processing,  and  subsequently  software,  functional  and  performance 
requirements  which  represent  the  primary  input  to  the  subsequent  process 
design  activity.  Requirements  from  the  system  specification  which  the 
Data  Processing  subsystem  must  perform  in  any  operational  situation  are 
allocated  to  hardware  and  software.  They  are  iterated  with  the  system 
level  requirements  to  assure  completeness  and  traceability. 

•Once  all  the  Data  Processing  subsystem  related  requirements  have 
been  identified,  the  generation  of  the  requirements  specification  is  the 
next  order  of  business.  Special  attention  must  be  given  during  this  acti- 
vity as  to  what  makes  a good  requirement  and  specification.  For  real- 
time systems,  the  Data  Processing  subsystem  functional  requirements  are 
best  expressed  in  terms  of: 

1)  Input 

2)  Processing  Requirements 

3)  Output 

That  is  for  each  external  stimulus,  the  subsystem  must  perform  cer- 
tain processing  functions  in  order  to  produce  the  desired  output.  Require- 
ments expressed  in  such  a manner  aid  in  assignment  of  performance  require- 
ments, establish  a one-to-one  correspondence  between  requirements  and 
interface  specifications,  facilitate  the  verification  of  requirements  and 
preliminary  design,  and  facilitate  testing. 

Equally  important  are  the  properties  a specification  should  have. 

The  attributes  which  are  most  important  are  completeness,  consistency, 
traceabi 1 i ty , and  testability.  By  careful  definition  of  Data  Processing 
subsystem  requirements  adhering  to  the  general  attributes  described  above, 
the  resultant  requirements  specification  will  enhance  software  design 
soluti on . 
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Interface  definition  parallels  the  Data  Processing  Subsystem  require- 
ments generation.  Since  the  processing  requirements  are  organized  by 
stimuli,  it  is  absolutely  essential  that  for  each  external  interface  a 
complete  message  catalog  be  established.  Furthermore,  compatibility  bet- 
ween interface  and  functional  requirements  must  be  assured;  that  is,  the 
one-to-one  correspondence  discussed  above. 

Once  the  Data  Processing  Subsystem  functional  requirements  have  been 
identified  and  documented  in  a traceable  fashion,  they  must  be  analyzed 
and  verified  for  their  adherence  to  the  critical  attributes.  The  require- 
ments analysis  group  utilizes  a formalized  method  of  describing  require- 
ments in  terms  of  paths.  The  requirements  are  functionally  grouped  and 
then  decomposed  in  flow  chart  forms  which  illustrate  the  interaction 
between  them.  These  flow  charts,  called  Requirements  Networks  or  R-NETS, 
consist  of: 

1)  A set  of  nodes  representing  processing  steps  called  alphas. 

2)  A set  of  nodes  identifying  processing  conditions,  called  "0R- 
nodes". 

3)  A set  of  nodes  identifying  "don't  care"  sequences  of  alphas, 
which  could  be  performed  in  any  sequence  or  even  in  parallel 
("AND-nodes"). 

4)  A set  of  interface  nodes  where  external  messages  are  received 
or  transmitted. 

5)  A set  of  initial  and  terminal  nodes. 

6)  A set  of  arcs  joining  the  nodes  to  form  a network. 

7)  A set  of  events  and  conditions  for  R-NET  enablements. 

8)  A set  of  validation  points  used  to  uniquely  identify  transitions. 

Each  alpha  is  associated  with  two  subsets  of  data  identifiers  which 
represent  data  input  to  and  output  from  the  processing  step.  These, 
together  with  the  definition  of  the  interface  data  and  the  identification 
of  the  choice  variables,  form  the  basis  for  consistency  checking  of  the 
paths.  On  the  other  hand,  if  an  R-NET  processes  data  from  an  interface, 
the  net  shows  all  possible  processing  paths  emanating  from  the  interface. 
Thus,  the  processing  of  each  message  for  each  possible  condition  is 
identified. 
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This  provides  a significant  aid  in  verifying  completeness  of  require- 
ments. 

Performance  requirements  are  assigned  to  validation  points  pro- 
viding a connection  between  these  and  processing  requirements.  Each 
validation  point  is  associated  with  a set  of  data  identifiers.  If  per- 
formance requirements  are  expressed  in  terms  of  these  identifiers,  then 
the  requirements  will  be  testable. 

A collection  of  R-NETS  define  a set  of  Data  Processing  functional 
requirements  which  can  be  checked  for  completeness,  consistency  and 
testabl il ity. 

A functional  simulator  is  utilized  to  further  analyze  and  validate 
the  requirements.  Simulators  model  the  functional  characteristics  of  the 
candidate  Data  Processing  Subsystem  and  the  way  its  resources  are  utilized. 
Significant  stimuli,  hardware  characteristics  as  well  as  critical  pro- 
cessing paths  from  the  R-NET  are  modeled. 

Via  functional  simulation,  the  requirements  analysis  team  can 
demonstrate  that  the  given  Data  Processing  subsystem  requirements  are 
implementable  on  the  selected  hardware  configuration.  In  the  event 
incompatibility  exists,  system/subsystem  trade-off  studies  are  conducted 
until  a viable  baseline  is  established.  An  important  by-product  of  the 
use  of  the  functional  simulation  is  that  it  provides  another  method  by 
which  the  viability  and  compatibility  of  the  Data  Processing  Subsystem 
requirements  are  demonstrated. 

Once  a viable  hardware  configuration  has  been  established,  the 
requirements  are  allocated  to  hardware  and  software.  It  is  important  that 
a complete  traceability  be  maintained  from  the  system  to  the  hardware 
and  software  specifications.  Interface  specifications  are  also  established 
at  this  point  of  the  preliminary  design  phase.  From  the  message  catalog 
defined  earlier,  interface  specifications  are  generated  which  define 
functional  requirements  and  quality  assurance  provisions  for  each  Data 
Processing  Subsystem  interface.  They  specify  communications  in  terms  of 
external  messages,  format,  rate  and  the  data  transfer  protocol  applicable 
to  the  interface. 
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Process  Design:  The  structuring  of  functional  requirements  and  operational 
logic  into  distinct  software  functions  prior  to  allocation  to  software 
units,  and  the  performance  of  special  tasks  at  the  overall  software  system 
level  such  as  the  estimation  of  timing,  accuracy  and  storage  requirements, 
the  evaluation  of  processing  limitations,  and  the  analysis  of  system  be- 
havior and  computational  loading  characteristics.  Design  is  performed  at 
the  process  level  where  the  objective  is  to  ascertain  that  the  entire 
process  fits  together  and  can  perform  the  required  tasks  within  the  given 
time  limits  and  accuracy  requirements,  that  all  interfaces  are  properly 
specified,  and  that  the  data  base  is  correctly  defined.  Subpractices 
within  process  design  include  the  allocation  of  software  requirements  to 
software  units,  the  specification  of  all  data  exchanges  via  N charts,  a 
functional  simulation  to  analyze  timing  and  loading  behavior,  a preliminary 
sizing  of  all  data  processing  resources,  an  identification  of  critical 
issues  and  process  limitations,  a specification  of  the  execution  sequence 
for  all  logic  threads. 

Process  design  is  the  activity  that  establishes  a candidate  pre- 
liminary design  baseline.  The  three  major  inputs  to  this  design  activity 
are  the  baselined  and  validated  Data  Processing  Subsystem  functional  and 
interface  requirements  and  the  candidate  Data  Processing  hardware  char- 
acteristics. 

The  first  step  in  top-down  process  design  activity  is  the  "mapping" 
of  functional  and  interface  requirements  to  operating  system  and  applica- 
tion program.  These  requirements  are  then  further  allocated  downward  into 
software  architectural  elements.  The  purpose  of  this  mapping  is  to  define 
the  required  operating  system  support  services  and  to  subdivide  the 
application  program  into  a number  of  independently  schedulable  modules 
which  perform  specific  major  processing  functions. 

Along  with  the  definition  of  the  software  architectural  elements, 
the  definition  of  the  process  structure  which  will  best  meet  given  per- 
formance requirements  is  required.  This  activity  defines  how  the  process 
(which  consists  of  the  operating  system  and  application  programs)  will 
t,  perform  as  a result  of  external/internal  stimuli.  Software  analysis  de- 

termines intermodule  communications,  scheduling/dispatching  criteria. 
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priority  servicing,  load  handling,  error  processing  and  recovery.  The 
impact  of  external  intermodule  communications  upon  the  data  base  structure 
is  folded  into  the  data  base  design  activity. 

Once  the  modules  of  a program  have  been  defined,  functional  and  inter- 
face requirements  are  further  allocated  downward  to  the  lowest,  meaningful 
entities  of  the  software  structure;  that  is,  to  routines. 

Two  important  guidelines  are  followed  in  this  series  of  downward 
allocation  of  requirements.  Firstly,  complete  traceability  of  requirements 
must  be  maintained  to  the  routine  level.  A traceability  matrix  must  be 
maintained  to  the  routine  level.  That  is,  a traceability  matrix  must  be 
generated  which  identifies  the  allocation  of  each  software  and  interface 
requirements  paragraph  (subparagraph)  to  software  modules  and  routines.  The 
second  important  guideline  is  that  performance  budgets  of  storage,  accuracy 
and  timing  must  also  be  allocated  down  to  the  module  and  routine  level.  These 
budgets  are  derived  from  software  performance  specification,  hardware  con- 
straits  and  software  analysis.  Their  judicious  allocation  to  software 
architectural  elements  can  assure  that  software  performance  requirements 
are  controllable  within  the  given  hardware  limitations. 

An  important  activity  which  supports  the  generation  of  realistic 
routine  budget  estimates  is  the  key  algorithm  design.  This  design  activity 
is  conducted  to  solve  critical  areas  before  detailed  design  and  production 
coding  begins.  Candidate  critical  areas  are  identified  with  as  much  detail 
as  possible  during  process  design.  Prototype  software  is  developed  to  find 
appropriate  techniques  for  solutions. 

The  data  base  design  is  also  accomplished  as  part  of  the  top-down 
process  design  activity.  The  top-level,  functional  data  base  is  defined 
using  N^-chart.  For  a process  designer,  this  chart  provides  a top-level 
base  from  which  data  flow  and  design  modularity  for  the  entire  Data  Pro- 
cessing Subsystem  can  be  assessed.  Utilizing  the  N^-chart,  the  global 
data  base  is  defined  by  the  process  designers  in  terms  of  data  files.  The 
preliminary  data  base  design  activity  establishes  the  general  category  of 
the  data  base  parameters  for  each  file  with  preliminary  definition  of  the 
data  base  constants  and  variables,  formats,  units,  etc.  Size  of  data  files 
are  estimated  and  fed  back  to  the  budget  allocation  process  for  modules 
and  routines. 
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In  summary,  the  process  design  allocates  Data  Processing  Subsystem 
requirements  and  interface  requirements  in  a traceable  manner  to  hardware 
and  various  programs  that  comprise  the  entire  software.  Furthermore,  the 
software  architecture  is  defined  and  requirements  and  performance  budgets 
allocated  to  the  module/routine  level.  The  process  interaction  and  control 
have  been  established  and  data  base  structure  designed.  Consequently,  a 
candidate  development  baseline  has  been  established. 

Incremental  Development:  The  identification,  design,  coding  and  testing 
of  functionally  meaningful  subsets  (i.e.,  increments)  of  the  overall  soft- 
ware system  wherein  each  increment  1)  provides  a self  sufficient,  execu- 
table and  testable  portion  of  the  complete  software  capability  and  2)  adds 
to  and  builds  upon  the  preceding  increment.  Incremental  development  is  an 
overall  software  production  approach  involving  the  definition  and  implemen- 
tation of  progressively  more  complete  increments  (or  "loops")  of  system 
capability.  Each  increment  is  self  sufficient  and  provides  an  independently 
testable  increment  of  the  evolving  program  containing  real  deliverable  code 
as  well  as  dummy  code  for  those  functions  that  are  to  be  fully  developed 
in  subsequent  increments.  The  final  increment  includes  all  the  deliverable 
code  as.  a minimum.  Builds  within  each  increment  carry  the  incremental 
development  approach  to  more  detail  and  further  reduce  the  chance  of  down- 
stream problems.  Incremental  development  reduces  risk  by  developing  critical 
software  first,  evaluating  system  performance  gradually  and  allowing  an 
early  start  to  test  and  integration  activities. 

Continuation  of  top  down  design  concepts  in  conjunction  with  an 
incremental  implementation  approach  is  accomplished  through  identification 
and  development  of  a dummy  process  to  begin  with,  followed  by  successive 
updates  replacing  dummy  software  modules  with  actual  code.  This  incremental 
approach  is  supported  by  earlier  development  of  the  overall  software  struc- 
ture during  preliminary  design.  Within  that  structure  it  is  possible  to 
identify  selected  portions  of  the  software  (e.g.,  processing  threads  in- 
volving critical  functions)  which  can  be  developed  and  tested  as  independent 
increments.  With  care,  these  increments  (or  loops)  can  be  chosen  to  provide 
a complete  increment  of  system  capability  within  which  high  visibility 
testing  of  critical  functions  can  be  achieved  with  relative  ease.  Each 
completed  increment  thus  represents  not  only  1)  a meaningful  version  of 
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the  evolving  system  containing  more  capability  than  the  previous  increment 
but  also  2)  a testbed  (including  the  process  structure  and  test  information) 
with  which  interface/integration  testing  of  subsequently  completed  modules 
can  be  done. 

Complete  Preliminary  Design:  The  definition,  documentation  and  baselining 

of  a complete  preliminary  design  prior  to  detailed  design  and  coding,  and 
including,  as  a minimum,  the  top-level  software  structure,  data  base  def- 
inition, interface  characteristics,  scheduling  criteria,  and  analysis  of 
critical  algorithms.  The  preliminary  design  specification  contains  all  the 
conceptual  design  information  needed  to  support  the  detailed  definition  of 
the  individual  software  system  components  and,  upon  completion  of  the 
Preliminary  Design  Review,  becomes  the  work  statement  and  design  baseline 
for  development  of  the  detailed  design  specification  used  in  coding.  The 
preliminary  design  specification  has  a previously  agreed  upon  standard 
format,  identifies  the  mapping  from  requirements  to  software  design, 
identifies  the  organizations  responsible  for  the  software  to  be  developed, 
defines  the  software  structure,  specifies  required  processing  resources, 
specifies  the  data  base  organization  and  makes  uses  of  the  results  fur- 
nished by  the  process  design  to  specify  the  estimated  resource  budgets. 

During  the  process  design  a candidate  preliminary  design  baseline 
is  established.  A significant  step  in  modern  programming  practice  is  the 
demonstration  that  the  preliminary  design  baseline  is  viable.  The  software 
preliminary  design  baseline  is  v.onsidered  complete  when  the  following 
conditions  are  met: 

1)  Ail  applicable  system  level  requirements  have  been  allocated  to 
the  software  structure. 

2)  Interface  control  has  been  defined  and  implemented  via  interface 
speci fications. 

3)  Allocation  of  stated  software  and  interface  requirements  into 
basic  software  structural  elements  has  been  completed. 

4)  Storage,  execution  timing  and  accuracy  budgets  have  been 
established  for  each  software  module/routine. 

5)  The  data  base  has  been  defined  in  sufficient  detail  to  demonstrate 
software  operability. 
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6)  Satisfactory  design  approach  for  the  entire  software  process 
and  its  elements  has  been  demonstrated. 

7)  Software  performance  requirements  can  be  met  under  required 
operational  loading  conditions. 

Primary  methods  of  demonstrating  the  above  points,  that  is  the 
completeness  of  preliminary  design,  are  the  use  of  functional  simulation, 
thread  analysis  and  preliminary  design  review. 

The  functional  simulator  models  the  characteristics  of  the  Data 
Processing  hardware,  external  stimuli,  operating  system  and  each  module 
of  the  application  program.  Each  module  is  modeled  by  its  service  requests 
(data  and  operational)  to  the  operating  system,  its  logical  structure, 
the  execution  time  for  each  segment  in  the  logic  structure  and  data  base 
interaction  (both  external  and  internal).  Performance  budgets  established 
during  process  design  are  utilized  in  the  simulator.  This  detailed  level 
of  modeling  allows  an  evaluation  and  insight  into  the  software  preliminary 
design  and  its  complex  interaction.  The  simulator  is  used  by  the  process 
designers  to  examine  the  sensitivities  of  computer  capacity,  timing 
responsiveness,  loading  conditions  and  port-to-port  timing.  Process  design 
issues  such  as  proper  allocation  of  requirements  to  software  modules, 
data  base  interaction  efficiency,  and  recording  capacity  can  be  studied. 
Expected  operational  conditions  are  simulated  to  verify  that  the  software 
can  meet  its  performance  requirements. 

Preliminary  design  validation  via  the  use  of  functional  simulation 
is  an  extremely  important  method  during  the  preliminary  design  review 
process  for  real-time  software.  This  review  provides  to  the  software 
designers  and  the  design  reviewers  confidence  that  tne  preliminary  design 
baseline  meets  the  software  and  interface  requirements,  that  compatibility 
exists  between  hardware/software,  that  the  design  is  a workable  solution 
and  that  it  meets  its  performance  requirements  under  required  operational 
conditions. 

Thread  analysis  is  another  method  of  demonstrating  the  completeness 
of  the  preliminary  design.  This  analysis  identifies  each  unique  input  to 
the  DP  Subsystem  and,  for  each,  a thread  is  generated  identifying  the 
applicable  software  module  involved  in  processing  this  input.  The  unique 
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software  requirements  which  are  involved  in  such  processing,  the  data 
base  interaction  (at  the  file  level)  associated  with  the  given  input 
processing  and  the  output  produced  are  identified  on  these  threads.  Port- 
to-port  timing  requirements  are  also  identified  and  are  verified  by  the 
use  of  the  functional  simulator. 

Thread  analysis  is  employed  by  the  process  designers  to  provide 
additional  assurance  that: 

1)  Each  unique  input  is  properly  processed  and  the  required  output 
is  produced. 

2)  The  requirements  are  complete. 

3)  All  requirements  have  properly  been  allocated  to  software  modules. 

4)  Inter-module  interactions  and  data  base  definition  are  complete. 

At  the  completion  of  the  preliminary  design  phase  a Preliminary 
Design  Review  is  conducted.  This  review  provides  the  media  by  which  the 
soundness  and  completeness  of  the  preliminary  design  can  be  demonstrated. 
Simulator  results,  thread  analysis  and  other  material  are  presented  to 
this  effect.  A successful  Preliminary  Design  Review  becomes  the  foundation 
from  which  software  detailed  design  and  code  production  can  emanate. 

Unit  Development  Polders:  The  generation,  maintenance  and  review  of  a 
document  that  is  an  integral  element  of  a "document  as  you  go  policy"  and 
that  1)  serves  as  a collection  point  for  all  pertinent  development  and 
test  information  (requirements,  design  data,  code,  flow  diagrams,  test 
plans,  test  results,  etc.)  for  each  identified  software  unit,  and  2) 
requires  completion  of  a cover  sheet  specifying  a detailed  development 
plan  (i.e.,  programmer  estimates  of  completion  dates  for  intermediate 
milestones)  and  providing  a means  for  indicating  actual  completion  dates 
and  review  events  to  depict  the  status  of  each  software  unit. 

A Unit  Development  Folder  (UDF)  is  prepared  and  maintained  for  each 
software  unit  in  order  to  provide  an  organized  accessible  collection  of 
all  requirements,  design  data,  code  and  test  data  pertaining  to  that  unit, 
as  these  data  are  produced,  and  to  collect  unit-level  schedules  and  status 
information.  The  folders  are  established  within  one  week  after  the  completion 
of  PDR  and  are  maintained  until  the  final  as-built  software  documentation 
is  delivered. 
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Each  UDF  contains  a cover  sheet  identifying  the  contents  and  schedule  of 
each  section  of  the  folder.  It  is  signed  off  by  the  originator  upon 
completion  of  each  section.  The  sections  of  the  UDF  are  briefly  described 
below: 

• Section  0 (Cover  Sheet  and  Schedule)  contains  a cover  sheet 
delineating  scheduled  due  dates,  actual  completion  dates, 
assigned  originators  and  reviewer  sign-offs  and  dates  plus  a 
change  log  recording  the  UDF  revision  history. 

t Section  1 (Requirements)  identifies  the  baseline  requirements 
specification,  enumerates  the  requirements  which  are  allocated 
for  implementation  in  the  software  unit,  and  shows  the  mapping 
(by  paragraph  number)  to  the  requirements  specification. 

t Section  2 (Design  Description)  contains  the  current,  working 
version  of  the  design  including,  at  the  appropriate  times,  the 
preliminary  design  and  successively  more  detailed  design  data 
and  flow  charts  suitable  as  a "code  to"  specification. 

• Section  3 (Functional  Capabilities  List)  contains  a list  of  the 
testable  functions  performed  by  the  unit  obtained  from  an 
independent  review  of  the  detailed  design  by  someone  other  than 
the  unit  designer. 

• Section  4 (Unit  Code)  contains  current  source  code  listings  for 
the  unit  and  a change  log  of  post-baseline  updates  to  the  unit 
code. 

• Section  5 (Unit  Test  Plan)  describes  the  overall  testing  approach 
and  each  test  case,  identifies  test  tools  or  drivers  used  for 
unit  testing,  and  illustrates  the  FCL/Test  Case  Matrix  relating 
test  cases  to  capabilities  tested. 

• Section  6 (Unit  Test  Plan  Review)  documents  the  findings  from 
an  independent  review  of  the  Plan  to  provide  assurance  that 
unit  test  cases  will  adequately  test  branch  conditions,  logic 
paths,  input  and  outputs,  and  error  handling  procedures. 
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• Section  7 (Test  Case  Results)  contains  a compilation  of  test 
case  results  and  analyses  necessary  to  demonstrate  that  the  unit 
has  successfully  passed  the  tests  prescribed  by  the  Unit  Test  Plan. 

• Section  8 (Problem  Reports)  contains  status  logs  and  copies  of 
all  Design  Problem  Reports,  Design  Analysis  Reports  end  Discrep- 
ancy Reports  which  document  design  and  code  changes  pertaining 
to  the  unit. 

• Section  9 (Notes)  contains  relevant  memoranda,  informal  reports 
and  other  notes  which  provide  supplementary  detail  expanding  on 
other  UDF  contents. 

Each  UDF  is  reviewed  by  the  assigned  reviewer,  other  than  the 
originator,  following  completion  of  each  UDF  section.  At  completion  of  unit 
testing,  the  manager  responsible  for  the  unit  reviews  the  entire  UDF  for 
technical  adequacy  and  completeness  and  signs  and  dates  the  cover  sheet 
to  indicate  approval.  In  addition,  periodic  audits  of  UDFs  are  conducted  to 
i ensure  compliance  with  project  standards. 

Enforced  Programming  Standards:  The  establishment  and  enforcement  of  strict 

, rules  to  direct  the  efforts  of  developers  in  the  production  of  code  and 

documentation  which  meets  or  surpasses  adopted  standards  involving  the 

1)  definition  of  certain  programming  practices  which  must  be  followed, 

2)  identification  of  other  practices  which  are  not  permitted,  and  3) 
establishment  of  an  active  review/inspection  procedure  to  ensure  compliance 
with  the  rules  and  achievement  of  the  standards. 

The  term,  "Programming  Standards",  generally  refers  to  both  1) 
approved  procedures  to  be  used  in  the  production  of  code  and  associated 
documentation  and  2)  minimum  acceptable  standards  against  which  the  completed 
products  are  measured.*  All  such  standards  are,  to  the  extent  possible t 
specified  early  in  the  project  and  documented  in  a Software  Standards  and 
Procedure  Manual.  One  of  the  primary  objectives  of  the  standards  is  to 

* Section  A. 1.2  provides  brief  descriptions  of  18  programming  standards 
f.  established  for  the  Systems  Technology  Program  software  development. 


guarantee  a high  degree  of  uniformity  across  all  individual  software 
system  components  which  a,  a to  be  integrated  and  must  interface  properly 
for  the  complete  system  to  work.  This  uniformity  is  especially  advantageous 
when  someone  other  than  the  original  developer  is  required  to  understand 
the  purpose,  function,  intricacies  and  limitations  of  a software  component 
during  integration,  test  and  maintenance  activities. 

Establishment  of  standard  programming  practices,  however,  is  only 
the  beginning.  If  the  standards  are  to  have  the  desired  effect,  they  must 
be  followed;  if  they  are  to  be  consistently  followed,  they  usually  must  be 
enforced.  Enforcement  o^strict  adherance  to  project  standards  is  accom- 
plished through  a combination  of  informal  and  formal  reviews  of  code  and 
documentation.  Performance  of  code  reviews  for  compliance  with  standards 
is  aided  substantially  by  source  code  scanners  including  the  Process 
Construction  Program,  Code  Auditor,  and  STRUCT.** 

Independent  Testing:  The  performance  of  formal  testing  by  a team  testing 
specialists  which  is  organizationally  separate  from  the  group  or  groups 
t responsible  for  software  design,  code,  debug,  and  development  testing.  It 

involves  analysis  of  software  requirements,  planning,  preparation,  and 
performance  of  tests,  review  of  test  results  for  satisfaction  of  require- 
ments, discrepancy  reporting,  and  retesting  after  discrepancies  are  re- 
solved and  fixes  are  made.  It  also  includes  support  to  the  developers 
through  review  of  design  documentation  and  designation  of  distinct  func- 
tional capabilities  to  be  exercised  during  development  testing. 

The  independent  test  group  determines  whether  or  not  the  software 
works  (not  how  it  works),  evaluates  requirements  and  design  to  assure  they 
are  testable,  prepares  comprehensive  test  plans,  supports  quality  assurance, 
and  plans  the  development  of  test  tools.  The  independence  of  the  test 
group  from  the  design  group  is  desirable  in  order  to  introduce  an  unbiased 
and  different  point  of  view  in  the  attempt  to  assess  the  capability  of  the 
developed  software. 

**  These  tools  are  described  below  in  Software  Development  Tools. 
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The  formal  testing  activity  is  briefly  characterized  as  follows: 

t The  primary  objective  is  to  verify  that  the  software  satisfies 
the  requirements  levied  upon  it. 

• The  testing  is  planned  and  directed  and,  in  general,  completely 
performed  by  the  independent  test  group. 

• Software  must  meet  specified  standards  prior  to  turnover  to  the 
test  group  (e.g.,  the  requirement  to  exercise  every  branch 
condition  in  a module  during  unit  level  testing). 

• Software  is  placed  under  formal  configuration  control  prior  to 
initiation  of  formal  testing  and  subsequent  changes  are  subject 
to  established  change  control  policies  and  procedures. 

t Formal  testing  is  audited  by  the  independent  quality  assurance 
group  for  sufficiency  and  compliance  with  test  plans,  procedures 
and  specifications. 

• Errors  are  documented  in  discrepancy  reports,  deficiencies  are 
analyzed  and  corrected  by  designers,  and  quality  assurance  audits 
are  performed  to  ensure  timely  and  satisfactory  problem  resolu- 
tion. 

• Test  tools  are  extensively  used  to  supplement  the  testor's  efforts 
by  completely  handling  otherwise  tedious,  error-prone  and  diffi- 
cult tasks. 

Software  Configuration  Management:  The  establishment  and  practice  of  for- 
mal software  management  principles  to  set  policies  and  procedures  for  the 
review  and  acceptance  of  contractual  and  internal  baselines,  and  including 
1)  configuration  identification  and  documentation,  2)  configuration  control 
to  monitor  changes  to  the  established  baseline  specification,  and  3)  con- 
figuration status  accounting  to  report  discrepancies  and  record  software 
implementation  status. 

Configuration  management  establishes  a series  of  baselines  and 
methods  of  controlling  changes  to  these  baselines  throughout  the  develop- 
ment and  test  phases.  A configuration  management  plan,  addressing  the  re- 
quirements of  this  practice,  is  prepared  by  the  project  and  approved  by 
the  project  manager  prior  to  the  first  software  design  review.  This  plan 
specifies: 
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• the  baseline  structure  to  be  used  by  the  project  (requirements, 
design,  test,  product) 

• the  baseline  definitions  (products,  events) 

• the  configuration  identification  ground  rules  including  the 
types  of  products  to  be  controlled  and  the  rules  for  identifying 
these  products 

• the  configuration  control  mechanisms  including  the  definition 
of  change  and  approved  processes  for  controlled  products  and 
the  handling  of  waivers  and  deviations 

• the  configuration  status  accounting  system,  including  the  records 
and  reports  required  to  assure  traceability  of  changes  to  con- 
trolled products  and  to  provide  a basis  for  communication  of 
configuration  information  within  the  project 

• the  configuration  verification  approach  including  periodic  audits 
to  assure  that  products  are  developed  and  maintained  according 

to  the  groundrules  defined  in  the  plan. 

The  configuration  management  activity: 

• establishes  and  operates  a product  development  library  in  which 
the  controlled  products  are  maintained 

• establishes  and  maintains  a regular  problem/discrepancy  reporting 
system 

• monitors  the  issuance,  retention,  change  control,  packaging  and 
delivery  of  the  computer  products  for  conformance  to  current 
baselines. 

Baselining  of  Requirements  Specifications:  The  review,  customer  acceptance 
and  documentation  of  the  software  requirements  specification  to  be  used  as 
a formal,  approved  baseline  for  software  development. 


The  baselining  of  requirements  occurs  after  the  preparation  of  a 
software  requirements  specification  and  prior  to  the  first  software  design 
review.  It  is  intended  to  achieve  a mutual  understanding  and  written  agree- 
ment as  to  proper  interpretation  of  mandatory  provisions  of  the  require- 
ments specifications  which  are  the  basis  for  software  end-item  acceptance 
prior  to  delivery. 
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Prior  to  baselining  the  software  and  interface  requirements,  it  is 
necessary  to  demonstrate  that: 

• The  Data  Processing  Subsystem  (DPS)  is  compatible  with  the  overall 
system  definition. 

• All  applicable  system  level  requirements  have  been  allocated,  in 
a traceable  manner,  to  the  DPS  and  subsequently  to  hardware  and 
software. 

• Software  requirements  are  complete,  consistent  and  testable. 

• Consistent  interface  control  has  been  established. 

• The  DPS  requirements  can  be  implemented  on  the  selected  hardware 
configuration. 

In  preparing  for  and  conducting  the  review  activities  that  precede  base- 
lining of  the  specification,  the  requirements  are  analyzed  and  evaluated 
for  technical  and  contractual  acceptability  and  any  identified  problems 
are  documented.  All  issues  and  problems  are  discussed  and  resolved  and  all 
agreements  and  action  items  resulting  from  the  review  are  documented  prior 
to  baselining  of  the  requirements  specification. 

Formal  Inspection  of  Documentation  and  Code:  A formal,  efficient  and  econom- 
ical method  for  conducting  independent  and  unbiased  review  and  assessment 
of  the  quality  of  software  documentation  and  code.  Includes  formal  inspection 
techniques  such  as  peer  group  critiques  and  walkthroughs,  quality  assurance 
reviews  and  audits,  and  design  reviews. 

Formal  inspection  of  documentation  and  code  facilitates  the  early 
detection  of  design  errors.  A design  walkthrough,  for  example,  is  accom- 
plished by  having  the  software  unit  design  reviewed  by  one  or  more  individ- 
uals other  than  the  originator.  The  scope,  technique  and  degree  of  formality 
of  the  walkthrough  is  established  by  the  project  manager  prior  to  PDR.  The 
review  includes,  as  a minimum,  checks  for  responsiveness  of  design  to 
requirements,  design  completeness  and  consistency,  flow  of  data  through 
input/output  interfaces,  error  recovery  procedures,  modularity,  simplicity 
and  testability.  The  technique  used  for  walkthroughs  consists  of  either  a 
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presentation  of  the  design  and  the  code  by  the  originator  in  the  presence 
of  the  reviewers,  or  a review  of  the  documentation  followed  by  a discussion 
of  review  comments  involving  all  participants.  Problems  detected  during  the 
walkthrough  are  identified  in  a written  summary  and  made  available  to  the 
managers  responsible  for  the  software. 

Independent  quality  assurance  audits  are  conducted  at  key  intervals 
during  the  development  and  evaluation  of  software  through  requirements 
definition,  design,  coding,  testing  and  operation.  These  quality  assurance 
audits  are: 

• System  Requirements  Audit;  To  determine  whether  planned  testing 
will  ensure  satisfaction  of  requirements  and  to  make  sure  that 
requirements  are  traceable  to  the  next  higher-level  specification. 

• Product  Specification  Audit;  To  determine  whether  product 
specifications  conform  to  established  standards  of  format, 
content,  completeness,  and  level  of  detail. 

• Interface  Verification  Audit;  To  examine  interface  requirements, 
interface  design  and  program  specifications  to  identify  and  re- 
solve interface  problems. 

• Product  Specification  (Engineering  Design  Document)  Audit;  To 
examine  detailed  design  specifications  and  the  data  base  defini- 
tion to  ensure  that  all  routine  input/output  is  properly  defined, 
the  design  data  is  current,  the  structure  of  flow  diagrams  con- 
forms with  project  standards,  and  explicit  traceability  exists 

to  higher-level  specifications. 

• Unit  Development  Folder  (UDF)  and  Code  Audit;  To  incrementally 
1)  audit  UDFs  as  they  are  prepared  in  verifying  that  changes  to 
unit  requirements,  functional  design  description  and  flowchart 
and  interface  definitions  are  current,  2)  manually  (plus  with  the 
automated  Code  Auditor)  scan  the  unit  source  code  for  compliance 
with  programming  standards,  and  3)  measure  thoroughness  of  unit 
level  testing  and  identify,  if  necessary,  additional  testing 
requirements. 
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• Pre-Turnover  Audit;  To  assess  the  adequacy  of  development  testing 
of  products  prior  to  their  internal  delivery  to  the  independent 
test  group  through  examination  of  the  products  and  test  results 
for  1)  actual  versus  expected  output,  2)  completeness,  content, 
organization  and  approval  status  of  UDFs,  3)  compliance  with 
programming  standards,  4)  testing  thoroughness  (e.g.,  execution 
of  every  branch  condition),  and  5)  assurance  that  all  reported 
discrepancies  have  been  corrected  and  tested. 

• Testing  Audit;  To  supplement  formal  testing  done  by  the  independent 
test  group  to  ensure  1)  established  configuration  management  pro- 
cedures are  followed,  2)  test  specifications  are  maintained  cur- 
rent, 3)  test  reports  are  properly  prepared,  4)  test  procedures 
explicitly  define  tests  to  be  conducted  and  test  results  comply 
with  acceptance  criteria  in  the  test  procedure,  and  5)  test  data 
package  contents  are  complete  and  comply  with  approved  formats. 

Software  Development  Tools:  The  planned  provision  and  usage  of  general  pur- 
pose and  specially  developed  support  programs  at  various  points  during  the 
software  development  to  assist  requirements  analysis,  software  design,  code 
generation,  debug  and  checkout,  formal  software  testing,  and  documentation 
in  a partially  or  fully  automated  fashion.  This  is  a general  practice 
involving  the  use  of  automated  tools  for  differing  purposes  and  at  various 
stages  throughout  the  software  development  process.  Prominent  examples  of 
software  development  and  test  tools  are: 

• Data  base  building  tools;  To  build  and  maintain  data  definition 
libraries  and  data  value  libraries,  check  data  definitions,  and 
supply  complete  data  specifications  to  all  using  software  modules. 

• System  construction  tools;  To  permit  use  of  a high-level,  special 
purpose  language  in  describing  the  desired  system  organization 
and  subsequent  use  of  a tool  to  automatically  merge  system  ele- 
ments into  a unified  whole.  Also  to  provide  an  illustration  of 
system  structure,  module  hierarchy  and  cross-references.  (The 
STP  Process  Construction  Language  and  Program  (PCL  and  PCP)  are 
system  construction  tools.) 
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• Code-checking  tools;  To  automatically  check  every  line  of  code 
in  the  system  to  1)  identify  and  report  on  instances  in  which 
project  programming  standards  have  not  been  followed,  2)  identify 
potential  singularities  such  as  division  by  zero,  3)  verify  units 
consistency  for  each  usage  of  each  parameter,  and  4)  identify 
parameters  that  are  used  before  set,  set  but  not  used,  etc.  (For 
example,  PCP  counts  the  executable  source  statements  in  a module 
to  check  for  compliance  with  the  routine  size  modularity  pro- 
gramming standard,  STRUCT  analyzes  the  detailed  logic  structure 

of  a module  to  check  for  compliance  with  the  structured  coding 
standard,  and  Code  Auditor  automatically  checks  every  line  of 
code  for  compliance  with  most  of  the  other  detailed  STP  pro- 
gramming standards.) 

• Test  Tools;  To  aid  generation,  accumulation  and  evaluation  of 
testing  information  and,  specifically,  to  1)  support  identifica- 

1 tion  of  problems  early  in  the  development  cycle,  2)  identify 

required  test  data  that  relates  requirements  to  software  cap- 
abilities and  ensures  thorough  testing,  3)  select  test  cases  to 
be  used  in  retesting  changed  software,  4)  check  actual  test  re- 
sults against  previously  obtained  or  expected  results,  and  5) 
drive  and/or  monitor  software  test  execution  and  measure  testing 
thoroughness.  (For  example,  the  Code  Auditor  and  Units  Consistency 
Analyzer  (UNIC)  "test"  code  for  standards  compliance  and  parameter 
usage  consistency;  the  Product  Assurance  Confidence  Evaluator 
(PACE)  analyzes  code,  identifies  overall  logical  structure, 
inserts  "test  hooks"  into  the  code,  and  subsequently  monitors 
test  execution  and  reports  on  the  extent  to  which  the  software 
structural  components  are  exercised;  the  Data  Reduction  and 
Report  Generator  (DRRG)  provides  essential  support  in  analysis 
of  voluminous,  detailed  data  logged  by  the  real-time  process  for 
subsequent  post  processing  and  test  results  evaluation). 
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A. 2 Definitions  and  Terminology 

This  section  contains  1)  definitions  of  the  thirty  characteristics 
(i.e.,  qualities  of  software  and  the  software  development  process)  used  in 
the  Programming  Standards  Impact  Evaluation  survey,  2)  descriptions  of  the 
twelve  "typical  problems  plaguing  software  development"  used  in  the  MPP 
Impact  Evaluation  survey,  and  3)  brief  definitions  of  other  MPP  study 
terminology. 

A. 2.1  Software  Characteristics 

The  following  definitions  of  the  thirty  software  characteristics 
used  in  the  Programming  Standards  Impact  Evaluation  survey  reflect  common 
usage  of  this  terminology  in  the  surveyed  environment. 

Requirements  Traceability:  The  ability  to  demonstrate  the  allocation, 
implementation  and  test  verification  of  the  individual  requirements  from 
the  requirements  specification  to  the  design  specification  and  to  the 
specification  of  test  requirements. 

Code  Auditability:  The  ability  to  audit  code  for  compliance  with  design 
specifications  and  any  pre-designated  coding  standards. 

Documentation  Auditability:  The  ability  to  audit  documentation  for  com- 
pliance with  a pre-designated  set  of  documentation  formats,  standards  and 
requirements. 

Personnel  Motivation:  The  interest  and  enthusiasm  of  personnel  to  accomplish 
the  required  work  in  a timely  and  effective  manner  with  due  regard  for  its 
quality  and  for  management  directions  and  requirements. 

Interface  Consistency:  The  operational  compatibility  of  software  modules 
with  each  other  and  with  the  external  software/hardware  system. 

Software  Integration:  The  process  of  combining  and  testing  individual 
software  units  into  larger  entities  for  the  purposes  of  demonstrating 
interface  compatibility  and  testing  for  satisfaction  of  higher  level 
requirements. 

Documentation  Rework:  The  revision  of  documentation  to  reflect  updates  or 


to  correct  errors. 
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Coding  Rework:  The  revision  of  code  to  reflect  updates  or  to  correct  errors. 

Retesting:  The  execution  of  previously  run  successful  tests  to  assure 
consistency  and  non-degradation,  or  to  assess  the  impact  of  changes  made 
to  the  product. 

Customer/Con tractor  Relationship:  The  state  of  the  environment  that  exists 
between  the  customer  and  the  contractor. 

Testing  Thoroughness:  A measure  of  the  thoroughness  of  testing  based  on 

some  established  criteria,  e.g.,  percentage  of  code  executed;  percentage 
of  requirements  tested;  percentage  of  capabilities  demonstrated,  etc. 

Documentation  Understandability/Readabi lity:  The  relative  ease  with  which 
a program  document  may  be  read  and  understood  by  someone  who  has  a general 
familiarity  with  the  subject. 

Code  Understandability/Readabi lity:  The  relative  ease  with  which  program 
code  may  be  read  and  understood  by  someone  who  understands  the  language 
and  has  a general  familiarity  with  the  subject. 

Documentation  Maintainabil ity/Useabil ity:  The  relative  ease  with  which  a 
program  document  may  be  updated  and  used  in  tracking  changes  to  the  program 
code.  This  is  a function  of  both  the  use  of  documentation  tools  and  the 
format  and  standards  used  in  the  documentation. 

Code  Maintainabil ity/Useabil ity:  The  relative  ease  with  which  program  code 
may  be  updated  and  used  in  the  development  process.  This  is  a function  of 
both  the  design  and  programming  standards  imposed. 

Testabi 1 i ty:  The  degree  to  which  code  may  be  tested  and  test  results  evalu- 
ated under  conditions  approximating  its  destined  operational  environment 
and  the  degree  to  which  satisfaction  of  requirements  may  be  verified. 

Operational  Reliability:  The  probability  that  the  software  will  satisfy 
the  stated  operational  requirements  for  a specified  time  interval  or  a 
unit  application  in  the  operational  environment. 

Cost:  A measure  of  the  amount  of  resources  used  to  accomplish  a particular 

task. 

Schedule:  A plan  for  the  amount  of  elapsed  time  needed  to  accomplish  a 
particular  task. 


'r-'T.- 


. wnm  • 


r> 


PAGE  A 


26 


Consistency:  The  degree  to  which  separate  components  of  a system  abide  by 
the  same  standards  and  procedures  and  are  uniform  with  respect  to  format, 
content  and  interfaces. 

Completeness:  The  degree  to  which  the  design  and  documentation  define  and 
describe  all  of  the  essential  elements  of  a system. 

Documentabi 1 i ty:  The  ease  with  which  readable  and  correct  documentation 
can  be  produced. 

Codability:  The  ease  with  which  a design  solution  can  be  coded. 

Documentation  Error  Freguency:  The  number  of  errors  per  unit  measure  of 
documentation. 

Coding  Error  Frequency:  The  number  of  errors  per  unit  measure  of  code. 

Portabi 1 ity:  The  relative  independence  of  a software  product  from  the 
' computer  system  on  which  it  is  designed  to  operate. 

i Execution  Time:  The  amount  of  computer  time  actually  used  in  executing  a 

particular  set  of  functions. 

' Core  Requirements:  The  amount  of  immediate  access  memory  required  to  contain 

the  code  and  data  for  a particular  set  of  functions. 

Logical  Organization:  The  clear  and  progressive  delineation  of  flow  and 
function  in  design  and  code. 

Programmer  Productivity:  The  amount  of  time  and  resources  needed  to  produce, 
test  and  document  a given  unit  of  code. 

A. 2. 2 Typical  Software  Development  Problems 

Cost  Overrun:  Exceeding  the  contractual  price  for  accomplishing  a stated 
amount  of  work  or  exceeding  one's  allocated  budget. 

Development  Status  Invisibility:  The  inability  to  gauge  the  amount  of  work 
performed  for  the  amount  of  resources  expended  or  the  progress  being  made 
relative  to  the  schedule. 

Unreliability:  The  inability  o'  the  software  to  perform  in  accordance  with 
its  functional  requirements  or  its  unavailability  for  use. 
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Unma i ntai nabi 1 i ty : The  relative  difficulty  of  debugging  and  correcting 
software  errors  or  of  making  minor  modifications  or  adding  new  capabilities 
to  a delivered  product. 

Inadequate  Satisfaction  of  Real  Requirements:  The  inability  of  the  software 
to  perform  in  an  operational  environment  in  accordance  with  how  it  was 
envisioned  based  on  the  stated  requirements. 

Inefficient  Use  of  Resources:  The  undue  expenditure  and  waste  of  manpower 
or  computer  resources  in  accomplishing  a given  task. 

Schedule  Overrun:  Exceeding  the  contractual  time  for  accomplishing  a stated 
amount  of  work  or  exceeding  one's  allocated  time  budget. 

Inadequate  Planning  and  Control:  The  lack  of  visibility  and  control  over 
budgets,  schedules,  staffing  or  resource  allocation  and  performance. 

Project  Mismanagement:  Failure  of  project  management  to  allocate  and  control 
resources  necessary  to  achieve  the  desired  results  within  the  negotiated 
l budget  and  schedule. 

Lack  of  Programming  Discipline:  The  non-existence  or  non-adherence  to  a 
t software  development  methodology  containing  standards  for  software  design, 

coding,  documentation,  testing,  configuration  management  and  quality 
assurance. 

Lack  of  Conclusive  Testing:  The  non-existence  or  non-adherence  to  a 
documented  test  plan  and  procedures  which  adequately  define  the  test 
environment,  objectives  and  acceptance  criteria  or  adequately  trace  test 
cases  and  results  to  requirements  and  capabilities. 

Poor  Documentation:  The  inadequacy  of  the  documentation  to  describe  the 
design,  contents  and  use  of  the  software  at  the  level  required  for  maintenance 
and  operational  application. 

A. 2. 3 Other  Terminology  Used 

Pi screpancy : Any  of  a defined  set  of  problems  or  conditions  which  require 
the  generation  of  a problem  report  and  resultant  action  independent  of  its 
^ severity. 
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Executable  Instruction  (Statement):  A computer  instruction  or  statement 
which  requires  action  by  the  central  processing  unit  (CPU). 


Failure:  The  result  of  encountering  an  error  of  the  type  that  causes  a 
computer  program  to  abort,  halt  or  perform  inadequately. 


Line  of  Code:  An  individual  line  of  a code  listing  including  executable 
statements,  commentary  and  other  non-executable  statements  or  directives. 


Module:  A collection  of  one  or  more  software  entities  which  perform  a 
discrete,  definable  function. 


Practice:  A preferred  or  advisable  approach  or  method  which  is  acknowledged 
to  be  generally  beneficial  or  useful  in  achieving  a desired  result. 

Process : A collection  of  one  or  more  application  programs  and  their  data 
bases  and  the  operating  system  configured  for  a particular  haraware  con- 
figuration and  operational  objective. 

Routine  (Subroutine):  A set  of  instructions  and/or  statements  that  exist 
as  an  identifiable  entity  and  carry  out  some  well  defined  operation  or  set 
of  operations.  A routine  may  call  or  is  callable  by  other  routines. 

Standard:  An  unambiguous,  explicit  description  of  a desired  property  of  a 
product  or  a mandatory  practice  to  be  followed. 

Subsystem:  An  assemblage  of  components  which  operate  as  part  of  a total 

system  and  which  is  collectively  capable  of  performing  a specific  function 
within,  and  as  defined  by,  that  system. 

Task:  The  basic  unit  of  software  capability  dispatched  by  the  operating 
system,  consisting  of  a task  control  routine  and  the  required  number  of 
lower  level  routines. 


Unit  of  Code:  A manageable  element  of  software  architecture  containing 
certain  characteristics  and  encompassed  by  a Unit  Development  Folder. 


F, 
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APPENDIX  B 

STP  CHRONOLOGY  AND  ENVIRONMENT  DATA 

B. 1 Vendor  Supplied  Support  Software 

Table  B-I  lists  major  components  of  the  development  support  software 
provided  by  CDC  to  support  Systems  Technology  Program  software  development. 
Each  component  is  identified  by  name  and  briefly  described. 

B.2  STP  Development  Chronology 

Tables  B-II  through  B-IV  present  lists  of  significant  milestone  events 
in  the  development  of  major  STP  software  system  components: 

. • Tactical  Applications  Program  (TAP)  Table  B-II 

i • Tactical  Operating  System  (TOS)  Table  B-III 

(including  the  early,  non-real  time, 
functional  TOS  (FTOS)) 

< 

• Test  Support  Software  (TSS)  Table  B-IV 

B.3  MPP  Chronology 

Table  B-V  identifies  distinct  practices  that  have  been  part  of  the 
overall  STP  software  development  methodology  during  the  period  from  March, 
1972  to  the  present.  Most  were  conceptually  defined  as  elements  of  the 
methodology  at  the  contract  outset,  but  many  were  either  not: 

• sufficiently  refined  for  broad  application, 

• formally  established, 

• adequately  supported  with  available  tools,  or 

• appropriate  for  the  current  project  phase 

until  later  in  the  development  process.  Table  B-V  therefore  contains  two 
sets  of  dates  to  illustrate  1)  when  each  practice  became  part  of  the  STP 
development  methodology,  and  2)  when  each  practice  became  sufficiently 
well  defined,  formally  established,  supported  and  appropriate  for  use  by 
STP  developers.  An  asterisk  (*)  is  used  to  denote  "Before  March  1972". 


• m*  & , «»'•  *ni  . . Vida  ► . ‘ - - 


PAGE  B - 2 

TABLE  B-I. 

CDC  Vendor-Supplied  Development 
Support  Software 

Name 

Descri ption 

SCOPE  2.1 

CDC  7600/7700  Operating  System 

SCOPE  3.4 

CDC  6400/CYBER  Operating  System 

FTN  Library 

Math  library  for  FTN  compiler 

FORTRAN  Debug 

Debug  package  for  use  with  FORTRAN  programs 

COMPASS  2.0 

Assembly  language  assembler 

COMPASS  3.0 

Assembly  language  assembler  with  improved 
MACRO  and  MICRO  capabilities 

SYSMOD 

Modifies  the  system  nucleus  library  to  add, 
delete,  or  replace  system  overlays  and  inter- 
rupt handlers 

SYSLIBE 

Modifies  the  system  libraries 

SIF  Analysis  Program 

Analyzes  System  Information  File  (SIF) 

SYSDUMP 

Dead  dump  program 

SMM 

System  Maintenance  Monitor  (SMM)  to  support 
hardware  debug 

UPDATE/L IBEDIT 

Utility  for  source  deck  maintenance 

SORT/MERGE 

Utility  for  sort/merge 

Record  Manager 

Utility  for  I/O  operations 

Loader 

Load  user  programs 

Permanent  File  Manager 

Utility  for  maintaining  permanent  files 

SAS 

CDC  7600  CPU  Simulator 

PPL)  Simulator 

CDC  7000  PPU  Simulator 
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Table  B- II.  Tactical  Applications  Program  (TAP) 
Development  Chronology 


Begin  Engagement  Software  Design  and 
Development 

Loop  1: 

Preliminary  Design  Complete 
Detailed  Design  Complete 
Code  and  Checkout  Complete 
Development  Test  Complete 
Loop  2: 

Preliminary  Design  Complete 

Detailed  Design  Complete 

Code  and  checkout  Complete 

Release  1 (CR-1)  or  Loop  3: 

Preliminary  Design  Complete 

Detailed  Design  Complete 

Code  and  Checkout  Complete 

Process  Integration  and  Evaluation 
Testing  Complete 

Preliminary  Release  of  CR-1  Software 
Final  Release  of  CR-1  Software 

Release  2 (CR-2): 

Preliminary  Design  Complete 

Detailed  Design  Complete 


06  March  1972 

01  February 
25  May  1973 
11  September 

01  May  1974 

02  July  1973 

15  February  1974 
01  November  1974 

22  August  1974 

15  January  1976 
01  August  1976 
01  January  1977 

16  August  1976 
01  January  1977 

01  October  1976 
01  January  1977 
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Table  B-III.  Tactical  Operating  System  (TOS/FTOS) 
Begin  Engagement  Programs 
Preliminary  TOS  Design  Complete 
Version  1 FTOS: 

Version  2 FTOS: 

Detailed  TOS  Design  Complete 
Version  3 FTOS: 

Version  4 FTOS: 

Version  1 TOS  (Preliminary  Capability): 


Development  Chronology 
06  March  1972 
20  March  1973 
27  July  1973 
31  May  1974 
26  July  1974 
01  October  1974 
01  April  1975 
01  May  1975 

14  June  1975 
01  Aigist  1975 

15  March  1976 
01  July  1976 

15  October  1976 


1 
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Table  B-IV.  Test  Support  Software  (SETS/KTSP)* 
Development  Chronology 


Begin  TSS  Programs  Design  and  Development 


06  March  1972 


SETS  Engineering  Requirements  Document  (ERD)  Release  01  September  1972 

KTSP  Engineering  Requirements  Document  (ERD)  Release  02  February  1973 


Preliminary  Design  for  KMR  SETS  Complete 


Preliminary  Design  for  KMR  SETS  Complete 


KTSP  Preliminary  Design  Review 


28  February  1973 
24  October  1973 
24  October  1973 


' 


SETS  Loop  1:  Detailed  Design,  Code  and  Test  Complete  01  May  1974 

SETS  Loop  2:  Detailed  Design,  Code  and  Test  Complete  01  November  1974 

SETS  Loop  3:  Deferred  Past 

Current  Contract 
End  Date 

KTSP  Release  1 (CR-1)  or  Loop  3:  Detailed  Design,  Code  01  January  1977 

and  Test  Complete 

KiiP  Release  2 (CR-2):  01  january  1977 


%■ 


* SETS  = System  Environment  and  Threat  Simulator 
KTSP  = KMR  Test  Support  Program 


Table  B-V.  Systems  Technology  Program  Modern  Programing  Practices  (MPP) 

Summary  of  Chronology 


Concept  Defined  Available/Used 


Process  Construction  Program  (PCP) 

Automated  Documentation  Tools 

Unit  Development  Folders  (UDFs)  Ma> 

Programming  Standards 

Quality  Assurance  Tools  Aug 

Structured  Programming  Apr 

Top-down  Design  Concept 

Independent  Test  Organization 

Incremental  Development  Approach 

Complete  Preliminary  Design 

Data  Processing  Subsystem  Simulation 
(DPSS) 

Automated  Production  Support  Facilities 
Discrepancy  Reporting  System 
Requirements  Analysis  and  Validation 
Process  Design 
Test  Tools 

Document-as-you-go  Concept 
Software  Configuration  Management 
Quality  Assurance  Audits 
Chart 


May  1972 
* 

August  1972 
April  1974 


January  1973 
★ 

Oct-Nov.  1972 
May  1972 
November  1973 
April  1974 
April  1973 


April -May  1973 


August  1973 


March  1973 
February  1973 


* Before  March  1972 
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APPENDIX  C 

MATHEMATICAL  ALGORITHMS 

The  following  are  detailed  descriptions  of  the  statistical  algorithms 
developed  for  the  analysis  of  the  survey  raw  data.  These  algorithms  have  been 
implemented  in  the  software  program  IMPACT.  A listing  of  IMPACT  is  included  in 
Appendix  D.  An  A x B matrix  of  practice  versus  characteristic  is  assumed. 

The  (i, j ) th  element  of  this  matrix  represents  a measure  of  the  influence  that 
practice  i has  on  characteristic  j.  Let  be  the  number  of  survey  responses 
determined  for  the  (i»j)th  element.  Since  each  response  is  a discrete  variable 
ranging  from  -2  to  +2,  a distribution  function,  d..,  can  be  specified  for  each 
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In  a similar  manner,  the  overall  effect  that  a practice  has  on  the  complete 

2 

set  of  characteristics  can  be  determined.  The  overall  mean  and  variance  ck 
for  practice  i are  given  by: 

A 

•1-E 

j=i 
A 

•?-L 

j=l 

where 

N-  = Total  number  of  responses  obtained  for  practice  i, 

A 

N.  N.. 

l < ij 

j=l 

The  mean  va?ues  for  each  practice  of  the  Programing  Standards  Survey  are 

depicted  by  the  "dots"  in  Figure  C-l.  The  shaded  areas  in  the  figure  indicate 

the  plus  and  minus  one  sigma  (a^)  variations  about  the  means.  As  seen  from 

the  figure,  all  practices  (or  standards)  have  a positive  mean  value  indicating 

that  on  the  average  the  standards  generally  have  an  overall  positive  influence 

on  the  complete  set  of  characteristics.  It  should  also  be  noted  that  the 

one  sigma  variations  about  the  mean  are  generally  rather  large.  Therefore,  even 

though  the  mean  is  positive,  it  cannot  be  concluded  with  high  confidence  that 

the  standard  actually  has  a positive  effect.  In  order  to  make  more  definitive 

statements  concerning  the  overall  influence  of  the  standard,  the  shape  of  the 

distribution  function  must  be  considered.  The  evaluation  of  the  distribution 

functions  will  be  discussed  in  detail  in  Section  C. 2. 

Another  consideration  of  interest  is  what  effect  the  complete  set  of 

practices  has  on  a given  characteristic.  In  a manner  analogous  to  equations 

2 

C.l-3  and  C. 1-4,  the  overall  mean,  u . , and  variance  a.  for  characteristic  j 

J J 

can  be  computed  as: 


Nij  ^ij/Ni 


E 

k-1 


(Xijk-  »i>‘ 
Ni- 
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(C. 1-4) 
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(C. 1-5) 


(C- 1-6) 


where  N.  is  the  total  number  of  responses  received  for  characteristic  j,  i.e., 

J 

* B 


"j  =2.  »tj 

i=l 


(C. 1-7) 


The  mean  value  of  the  responses  for  each  characteristic  as  well  as  the  one 
sigma  bounds  for  the  Programming  Standards  Survey  are  plotted  in  Figure  C-2. 

In  general,  the  set  of  standards  has  a positive  influence  on  the  characteristic. 
However,  some  standards  have  a slightly  negative  influence,  as  for  example,  on 
characteristic  18  ("Cost")  and  characteristic  19  ("Schedule").  As  previously 
noted,  the  large  variance  in  the  data  makes  high  confidence  conclusions  difficult 
unless  the  actual  distribution  function  is  examined.  An  algorithm  for  hypothesis 
testing  based  on  the  distribution  function  will  be  discussed  in  the  next  section. 

C. 2 Distributional  Algorithms 

Hypothesis  test  procedures  are  utilized  in  this  section  for  the  analysis 
of  the  IMPACT  survey  data.  Section  C.2.1  discusses  the  theoretical  foundations 
of  the  procedures  and  Sections  C.2.2  through  C.2.5  detail  the  implementation  of 
the  hypothesis  tests. 

C.2.1  Hypothesis  Testing 

Consider  an  experiment  consisting  of  independent  repeated  trials  whose 
outcomes  have  been  classified  into  two  categories,  called  "plus"  and  "minus". 

Let  the  probability  of  "plus"  be  denoted  by  p and  the  probability  of  "minus" 
be  denoted  by  q.  Such  an  experiment  is  said  to  consist  of  Bernoulli  trials  and 
is  subject  to  the  constraint  that  p + q = 1.  By  observing  N trials  of  this 
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experiment,  it  can  be  determined  (within  some  probability  measure)  whether  or 
not  the  probability  of  pluses  exceeds  the  probability  of  minuses,  i.e.,  whether 
or  not  p ^ .5.  This  can  be  accomplished  using  the  procedures  discussed  below. 

Let  the  first  hypothesis  (usually  termed  the  null  hypothesis  and  denoted 
by  HQ)  be: 


Hq:  p < .5  , 

and  the  alternative  hypothesis,  H^,  be: 

Ha:  p > .5  . 

Based  on  the  observation  of  N trials,  either  Hq  or  must  be  accepted  as  the 
"true"  hypothesis.  Let  Np  be  the  number  of  "pulses"  observed  during  the  N 
trials.  For  large  N,  Np  can  be  modeled  as  a Gaussian  random  variable  with  mean: 

u = Np 

and  standard  deviation 

a = [Np  (1  - p)]  1/2 

If  p = .5,  the  mean  and  standard  deviation  are  respectively: 
u = N/2  , 

a = sM/2.  . 

Figure  C-3  shows  pictorially  the  relationship  between  the  distribution  of  Np 
for  various  values  of  P . m 

From  the  figure  it  is  observed  that  the  larger  the  valu<**vPNp,  the 
more  likely  it  is  that  p > .5.  To  quantify  this  observation,  let 

Np  - (N/2) 

1 = y«7i 


(C.2.1) 
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When  p = .5,  Z has  a Gaussian  distribution  with  mean  zero  and  standard  deviation 
one.  It  can  then  be  shown  that  one  should  accept  if 

Z > k,  (C.2-2) 

where  k is  a measure  of  the  probability  of  error,  i.e. , k measures  the  probability 
that  one  will  accept  when  actually  Hq  is  the  true  hypothesis.  The  relationship 
between  k and  the  probability  of  error  is  shown  in  Table  C-I  where  condifence 
level  is  defined  as  confidence  level  = 100  (1  - probability  of  error). 

Combining  equations  C.2-1  and  C.2-2  and  rearranging  indicates  that  HA 
should  be  accepted  (with  the  confidence  associated  with  k)  if 


Np 

N 


_k 

2v/N 


Table  C-I.  Relationship  Between  k and  Confidence  Level 


k 

Confidence  Level  (%) 

-3.0 

0.13 

-2.5 

0.6 

-2.0 

2.0 

-1.5 

7.0 

-1.0 

16.0 

-0.5 

31.0 

0.0 

50.0 

0.5 

69.0 

1.0 

84.0 

1.5 

93.0 

2.0 

98.0 

2.5 

99.4 

3.0 

99.9 

For  example,  let  N « 36.  Table  C-II  gives  the  cutoff  value,  x,  for 
together  with  the  associated  confidence  C.  The  table  is  read  as  follows: 
if  |p  _>  x,  then  the  confidence  associated  with  choosing  is  given  by  C. 
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From  the  table  it  is  seen  that  if  Np/N  is  greater  than  .75,  one  may  choose 
and  have  extremely  high  confidence  that  the  "pluses"  are  truly  in  the  majority. 

So  far  confidence  levels  have  been  determined  only  if  is  accepted. 

In  an  analogous  manner,  the  probability  of  error  associated  with  choosing 
Hq  can  be  derived,  i.e.,  the  probability  that  HQ  is  accepted  when  is  in 
reality  the  true  hypothesis. 

If  Ha  is  rejected  (i.e.,  Hq  accepted),  then  Np/N  is  less  than  the  cutoff 
value  x.  This  implies  that  NM/N  > (1  - x)  where  Nn  is  the  number  of  "minuses" 
observed.  Thus  the  confidence  associated  with  accepting  Hq  is  obtained  from 
Table  C-H  as  the  confidence  associated  with  the  cutoff  value  (1  - x).  For 
example,  if 


f - x - .25  , 

then  one  has  very  low  confidence  (.13%)  if  is  accepted.  However,  if  Hq  is 
accepted,  the  confidence  that  a correct  decision  has  been  made  is  99.9%,  (the 
confidence  associated  with  the  1 - x = .75  entry  of  Table  C - I I ) - Thus  for 
this  data,  Hq  would  be  accepted  with  a high  level  of  confidence. 

The  data  in  Table C -II  will  be  utilized  in  the  following  sections  to 
determine  the  confidence  in  the  assertions  associated  with  the  analysis 


Table  C-ii.  Example  Confidence  Level  for  N 


Cut-off  Value  x 
0.25 


Confidence  Level  C % 


of  the  Impact  Evaluation  Survey  data.  The  assertions  will  be  of  the  form: 


"Practice  i has  a positive  (or  negative)  influence  on 
characteristic  j." 
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To  determine  if  the  assertion  is  supported  by  the  data,  a several  stage 
hypothesis  testing  procedure  is  utilized.  First  a hypothesis  test  is  con- 
structed to  determine  if  practice  i does  influence  characteristic  j.  This 
determination  is  described  in  the  next  section.  If  it  is  concluded  that  the 
practice  does  influence  the  characteristic.  Section  A. 2. 3 will  describe  the 
algorithm  to  determine  if  this  influence  is  positive,  negative,  or  inconclusive. 
The  overall  procedure  will  be  presented  in  summary  form  in  Section  A. 2. 4. 


: 


C.2.2  Assertion  Confidence 

For  each  practice-characteristic  pair  (as  well  as  the  overall  data  for 
both  practices  and  characteristics),  the  following  hypotheses  were  tested: 


Hq:  The  respondents  to  the  impact  evaluation  survey  were  in 

agreement  that  the  practice  i did  have  an  influence  on 
the  characteristic  j (i.e. , influence  rating  f 0). 


H^:  The  respondents  to  the  impact  evaluation  survey  were  in 

agreement  that  the  practice  i did  not  have  an  influence  on 
the  characteristic  j (i.e.,  influence  rating  = 0). 


These  hypotheses  were  tested  based  on  the  statistic  Np(i,j)  defined  as: 


NR(i,j) 


NQ(i  ,j) 


> 


where, 


NQ(i,j)  = Number  of  indifferent  (0)  responses  observed  for  the 


(i,j)th  practice-characteristic  pair,  and 

Total  number  of  responses  for  the  (i,j)th  practice 
characteristic  pair. 


Since  N.j  was  usually  in  the  30  to  40  range,  Table  C-II  can  be  utilized  to 
give  an  indication  of  the  confidence  level  associated  with  the  hypothesis 
test. 


PAGE  C 


11 


high  confidence  choice.  Values  of  NR(i,j)  between  .25  and  .75  are  in  an 
uncertainty  zone  where  neither  hypothesis  can  be  selected  with  high  confidence. 

For  presentation  of  the  survey  results  in  this  report  the  level  of  con- 
fidence is  denoted  by  the  following  terms: 

Strong  Assertion:  A very  high  level  of  confidence  that  the 

assertion  is  correct.  Cutoff  values  are  as  follows: 

ND ( i , j ) > .75  - highly  confident  that  H.  is  correct, 

i.e.,  the  itn  practice  does  not  influence  the  j L 
characteri Stic. 

ND(i,j)  < -25  - highly  confident  that  H is  true,  i.e.,  the 

K 0 

ith  practice  does  influence  (either  positively  or  negatively) 
the  jth  characteristic. 

Medium  Assertion:  Indications  are  that  the  practice  does  influence 

• the  characteristic.  However,  the  data  does  not  support  a clear 

^ cut  verification.  Assertions  will  be  termed  medium  when 

.25  < Nr  (i,j)  < .50. 

, Weak  Assertion:  The  practice  probably  has  only  a weak  (if  any) 

influence  on  the  characteristic.  The  majority  of  the  respondents 
gave  an  indifferent  rating  to  the  practice-characteristic  pair 
(i.e. , .50  < Nr  (i,j)  5 .75). 

C .2.3  Assertions  (Influence  Rating) 

Once  it  has  been  determined  that  practice  i does  have  an  influence 
(strong,  medium,  or  weak)  on  characteristic  j,  the  next  step  is  to  determine 
if  the  influence  is  positive  or  negative.  To  accomplish  this  the  following 
hypotheses  were  tested: 

Hq:  Respondents  to  the  survey  agreed  that  practive  i had.  a 

positive  influence  on  characteristic  j (i.e.,  influence 

rating  +1  or  +2). 

H^:  Respondents  to  the  survey  agreed  that  practice  i had  a 

f.  negative  influence  on  characteristic  j (i.e.,  influence 

rating  »•!  or  -2). 
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These  hypotheses  were  tested  based  on  the  statistic  RNN.  . , defined  as 

1 »J 


RNN(l.j)  • MjW), 


where , 

NN(i , j ) is  the  number  of  respondents  who  indicated  that  practice  i had 
a negative  (-1  or  -2)  influence  on  characteristic  j,  and  NR ( i , j ) is 
the  total  number  of  respondents  who  indicated  that  practice  i had  an 
influence  (-2,  -1,  +1,  or  +2)  on  characteristic  j. 

If  RNN(i,j)  exceeded  .75,  hypothesis  is  accepted  with  high  confidence.  On 
the  other  hand,  if  RNN(i.j)  was  less  than  .25,  HQ  was  selected.  If 
.25  < RNN(i,j)  < .75,  neither  hypothesis  can  be  selected  with  high  confidence 
and  the  result  of  the  experiment  is  inconclusive. 

For  strong  assertions  which  were  not  inconclusive,  an  additional  hypothesis 
test  was  performed  to  pinpoint  the  influence  rating.  This  additional  hypothesis 
test  was  not  utilized  for  medium  or  weak  assertions  since  their  confidence 
level  is  low.  For  strong  assertions  for  which  a positive  influence  was 
established  the  following  test  was  made. 

Hq:  For  the  respondents  who  agreed  that  practice  i had  a positive 

effect  in  characteristic  j,  the  majority  agreed  that  the  influence 
was  mtlijm  (+1). 

H^:  For  the  respondents  who  agreed  that  practice  i had  a positive 

effect  on  characteristic  j,  the  majority  agreed  that  the 
influence  was  strong  (+2). 

This  hypothesis  was  tested  by  utilizing  the  statistic 

N+2( i . j )/N+(i , j ) (C. 2-3) 

where, 

N+2 ( "• » J ) is  the  number  of  respondents  who  indicated  that  practice 
i had  a +2  effect  on  characteristic  j,  and 


t Si.  £££  OKgKffiW’W*  ■ 


N+  (i,j)  is  the  number  of  respondents  who  indicated  that  practice  i 
had  a positive  influence  (+1  or  +2)  on  characteristic  j. 

If  the  ratio  given  by  equation  C.2-3  exceeded  .5,  hypothesis  was  accepted, 
if  not,  Hq  was  chosen.  An  analogous  hypothesis  test  was  performed  for  strong 
assertions  for  which  a negative  influence  was  established. 

C . 2.4  Summary 

Table  C-III  provides  a summary  of  the  hypothesis  test  algorithm  in 
tabular  form. 
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APPENDIX  D 

IMPACT  EVALUATION/COMPARISON  TOOLS 


D.l  General 

Three  programs  were  developed  to  support  the  analyses  performed  in  the 
MPP  study.  This  appendix  provides  a brief  description  of  the  purpose  and 
function  and  a source  listing  for  each  of  the  programs: 

• IMPACT 

• MERIT 

• RANK 

These  programs  were  coded  in  the  FORTRAN  extended  language  for  use  on 
the  CDC  Cyber  74/174  (TRW/TSS)  computer  system. 

D.2  IMPACT 


D.2.1  Purpose 

The  IMPACT  program  was  required  to  analyze  impact  evaluation  survey 
response  raw  data  and  produce  composite  impact  assessments. 

D.2. 2 Input 

IMPACT  accepts  as  input  the  coded  influence  ratings  (+2,  +1,  0,  -1, 

-2)  contained  in  the  impact  evaluation  matrices  for  N survey  questionnaires. 

A rating  value  of  3 is  used  to  indicate  a blank  in  the  evaluation  matrix. 

D. 2.3  Function 

IMPACT  performs  the  quantitative  and  distributional  algorithms  described 
in  Appendix  C. 
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For  each  element  of  the  impact  evaluation  matrix,  IMPACT  examines  the 
input  ratings  from  the  N survey  questionnaires  and  derives  a response- 
frequency-distribution  (i.e.,  the  number  of  +2  responses,  +1  responses, 
etc)  and  a corresponding  rating-percentage-distribution.  The  mean  rating 
and  standard  deviation  are  also  computed.  IMPACT  performs  the  hypothesis 
test  (distributional  algorithm)  on  the  response-frequency-distributions 
and  derives  a composite  assertion  strength/influence  rating  for  each  matrix 
element.  In  addition,  IMPACT  computes  and  analyzes  summary  response- 
frequency-distributions  for  each  matrix  row  and  column  and  for  the  entire 
matrix  and  derives  corresponding  summary  measures  of  assertion  strength/ 
influence  rating. 

D.2.4  Output 

IMPACT  output  consists  of  the  response-frequency-distributions, 
percentages,  mean  and  standard  deviation  statistics,  and  the  composite 
assertion  strength/influence  ratings.  Examples  of  IMPACT  output  (MPP 
theoretical  results)  are  presented  in  the  sections  following. 

0.2. 4.1  Mean  and  Standard  Deviation 

IMPACT  determines  the  mean,  number  of  non-blank  responses,  and 
standard  deviation  (variance)  for  each  matrix  element  (i,j  pair).  The 
results  presented  in  Table  D-III  are  to  be  interpreted  in  the  following 
manner: 

a.  A row  index,  (i.e.,  "practice"  index),  is  printed  vertically 
along  the  page.  (The  relationship  between  the  practice  and 
its  index  is  presented  in  Table  D-I). 


b.  A column  index,  (i.e.,  "problem"  index),  is  printed  horizon- 
tally across  the  page.  (The  mapping  of  problem  versus  inde~ 
is  presented  in  Table  D- I I ) . 
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Table  D-I  Practice  Indices 

Index Practice  Identification 

1 Requirements  Analysis  and  Validation 

2 Process  Design 

3 Incremental  Development 

4 Complete  Preliminary  Design 

5 Unit  Development  Folders 

6 Enforced  Programming  Standards 

7 Independent  Testing 

8 Software  Configuration  Management 

9 Baselining  of  Requirements  Specification 

10  Formal  Inspection  of  Documentation  and  Code 

11  Software  Development  Tools 
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Table  D-II  Problem  Indices 


Problem  Identification 

Cost  Overrun 

Development  Status  Invisibility 

Unreliability 

Unmaintainability 

Inadequate  Satisfaction  of  Real  Requirements 

Inefficient  Use  of  Resources 

Schedule  Overrun 

Inadequate  Planning  and  Control 

Project  Mismanagement 

Lack  of  Programming  Discipline 

Lack  of  Conclusive  Testing 

Poor  Documentation 


Jh 


I 

KjrcQjPHQ  8L5  r*  ■ 
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For  example,  the  mean,  number  of  non-blank  responses,  and  variance  of  the 
"Process  Design"  practice  (row  index  of  2)  and  "Unreliability"  problem 
(column  index  of  3)  pair  is  shown  on  Table  D-III  to  be: 

★ ★ ★ ★ ★ 

* 1.24  * 

* 49  * 

* .74  * 

* ★ ★ * * 

It  should  be  noted  that  following  column  12  is  a column  containing 
the  sample  mean,  number  of  non-blank  responses,  and  sample  variance  summary 
data  for  each  row.  Similarly,  the  row  following  row  11  contains  summary 
data  of  each  column. 

D.2.4.2  Response-Frequency-Distributions  and  Percentages 

IMPACT  determines  the  response- frequency-di stributions  and  per- 
centages for  each  matrix  element  (i,j  pair).  The  results  presented  in 
« Table  D-IV  are  to  be  interpreted  in  the  following  manner: 

I a.  Row  and  column  indices  are  printed  with  the  same  meaning  as 

stated  in  D.2.4.1-a  and  D. 2. 4. 1 -b. 

‘ b.  For  each  i , j pair,  two  columns  of  five  numbers  each  are 

derived.  These  columns  have  the  following  interpretation: 


Column  1 Ratina  Freauencv 

Column  2 Rating  Percentage 

Number  of  Strong  Positive 
(+2)  influence  ratings 

Percentage  of  +2  responses 

A 

Number  of  Positive  (+1)  influence  ratings 

Percentage  of  +1  responses 

Number  of  No  (0)  influence  ratings 

Percentage  of  0 responses 

k 

Number  of  Negative  (-1)  influence  ratings 

Percentage  of  -1  responses 

I* 

Number  of  Stronq  Neqative 
(-2)  influence  ratings 

Percentage  of  -2  responses 

For  example,  the  distribution  of  responses  showing  the  effect  that 
the  practice  "Unit  Development  Folders"  had  on  the  problem  "Development 
Status  Invisibility"  is  obtained  as  follows:  "Unit  Development  Folders" 

has  a row  index  of  5 and  "Development  Status  Invisibility"  has  a column 
K index  of  2.  Column  1 of  entry  (5,2)  of  Table  D-IV  shows  that  the  following 

-j  distribution  of  responses  was  obtained  for  this  i,j  pair: 


-4 ' 
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Number  of  Responses  Responses 

46  Strong  Positive  (+2)  influence  rating 

6 Positive  (+1)  influence  rating 

1 No  (0)  influence  rating 

1 Negative  (-1)  influence  rating 

0 Strong  Negative  (-2)  influence  rating 

Similarly,  Column  2 of  entry  (5,2)  shows  that  the  percentage  distribution 
of  responses  is  as  follows: 


Percentage  of  Responses  Responses 

85  Strong  Positive  (+2)  influence  rating 

11  Positive  (+1)  influence  rating 

2 No  (0)  influence  rating 

2 Negative  (-1)  influence  rating 

0 Strong  Negative  (-2)  influence  rating 

The  13th  column  contains  summary  data  of  each  row.  The  12th  row  contains 
summary  data  of  each  column. 

D.2.4.3  Composite  Assertion  Strength/ Influence  Ratings 


IMPACT  determines  the  composite  assertion  strength/influence 
ratings  for  each  matrix  element  (i,j  pair).  The  results  presented  in  Table 
D-V  are  to  be  interpreted  in  the  following  manner: 

a.  Row  and  column  indices  are  printed  with  the  same  meaning  as 
stated  in  D.2.4.1-a  and  D. 2. 4. 1 -b. 

b.  For  each  i , j pair,  an  assertion  strength/ influence  rating 
indicator  is  derived.  The  formulation  of  this  indicator  is 
discussed  in  detail  in  Appendix  C.  Table  C-III  summarizes 
all  of  the  possible  rating  indicators. 

From  Table  D-V,  the  assertion  strength/influence  rating  indicator 
of  row  index  3 and  column  index  1 is  SP.  By  referencing  Table  D-I,  D-II, 
and  C-III,  the  following  can  be  concluded  about  the  SP  rating  of  i , j pair 
(3,1):  "We  can  assert  with  strong  confidence  that,  in  the  theoretical  case, 

the  practice  (Incremental  Development)  can  and  should  make  a positive 
contribution  toward  eliminating  the  problem  (Cost  Overrun)". 


TABLE  D-III.  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results 


PAGE  D - 7 


******************************** ****** *********** 


TABLE  D-III.  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results  (Continued) 


ABLE  D-III.  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results  (Continued) 
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TABLE  D- III-  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results  (Continued) 
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TABLE  D-III.  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results  (Continued) 


TABLE  D- III.  IMPACT  Output  - MPP  Survey  Mean  and  Variance  Results  (Continued) 
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TABLE  DtIV.  IMPACT  Output  - MPP  Survey  Cumulative  Results 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummulative  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Curnmul ati ve  Results  (Continued) 
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TABLE  D-1V.  IMPACT  Output  - MPP  Survey  Cumulative  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummul ati ve  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummul ati ve  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummulative  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummulative  Results  (Continued) 
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TABLE  D-IV.  IMPACT  Output  - MPP  Survey  Cummulative  Results  (Continued) 
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TABLE  D-V.  IMPACT  Output  - Assertion  Strength/Influence  Ratings 
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TABLE  D-V.  IMPACT  Output  - Assertion  Strength/Influence  Ratings  (Continued) 
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D . 2 . 5 Listing 

The  IMPACT  source  listing  is  presented  in  Table  D-VI. 

0.3  MERIT 
D . 3 . 1 Purpose 

The  MERIT  program  was  required  to  analyze  the  effect  of  non- 
uniform  weighting  of  the  response-frequency-distributions  (see  D.2.3)  in 
computation  of  summary  measures  of  assertion  strength/influence  rating  for 
individual  MPP  and  selected  MPP  in  combination. 

D.3.2  Input 

MERIT  accepts  the  response-frequency-distributions  from  the  IMPACT 
program.  MERIT  also  accepts  a list  of  row  weights  (0  or  1 to  indicate 
which  rows  are  to  be  included  in  the  computation)  and  column  weights 
(between  1 and  99  to  indicate  the  appropriate  factor  for  each  column  of 
distributions)  used  in  computing  the  weighted  summary  response-frequency- 
distributions. 

D.3.3  Function 

MERIT  performs  the  distributional  algorithm  described  in 
Appendix  C.  MERIT  uses  the  input  row  and  column  weights  to  factor  each  of 
the  response-frequency-distributions  from  the  IMPACT  program.  MERIT 
totals  the  factored  distributions  across  each  row  and  down  each  column  to 
obtain  weighted  summary  response-frequency-distributions  for  each  row  and 
column  and  adds  the  column  summary  distributions  to  obtain  a weighted 
summary  response-frequency-distribution  for  the  complete  matrix.  The 
hypothesis  test  (distributional  algorithm)  is  performed  in  deriving  a 
composite  assertion  strength/influence  rating  (i.e.,  a figure  of  merit) 
from  each  of  the  summary  distributions. 

The  MERIT  program  was  designed  for  interactive  use,  permitting  the 
analyst  to  input  varying  sets  of  row  and  column  weights  to  investigate  the 
figure  of  merit  for  various  combinations  of  MPP  in  the  context  of  different 
software  development  activities. 
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D.3.4  Output 

MERIT  output  consists  of  the  summary  response-frequency- 
distributions  with  corresponding  percentages  and  the  composite  assertion 
strength/influence  ratings  (figures  of  merit).  Examples  of  output  are 
contained  in  Appendix  E.  Examples  of  MERIT  output  (MPP  theoretical 
results)  are  presented  in  the  section  following. 

D. 3.4.1  MERIT  Output  - Unweighted 

MERIT  obtains  the  response-frequency-distributions  generated  by 
IMPACT  and  applies  weighting  to  the  distributions  thus  perturbing  the  row 
and  column  summary  distributions,  and  the  summary  response-frequency- 
distribution  for  the  complete  matrix.  The  output  presented  in 
Table  D-VII  presents  the  row  and  column  summary  data  from  MERIT  uniform 
weights  (i.e.,  all  row  weights  and  column  weights  equal  to  one).  The 
output  consists  of: 

The  row/column  ( "Practice"/"Problem")  indices 

The  row/column  identification  under  the  label  "Characteristic" 
and  "Practice" 

The  weight  applied  to  that  particular  row/column 

The  summary  assertion  strength/influence  rating  for  that 
particular  row/column  after  application  of  the  weights 

The  summary  response-frequency-distribution  and  percentage  for 
that  particular  row/column  after  application  of  the  weights. 

Located  on  the  last  page  of  Table  D-VII  is  the  weighted  result 
for  the  complete  matrix,  i.e.,  the  "Composite  Figure  of  Merit." 

D.3.4. 2 MERIT  Output  - Weighted 

Table  D-VIII  shows  the  results  of  applying  varied  weights  to  the 
data.  Note  the  changes  in  the  response-frequency-distributions  and  the 
assertion  strength/influence  rating  for  practice  number  9,  and  for  the 
composite  figure  of  merit.  Another  example  of  weighting  with  MERIT  is 
discussed  in  Section  4 and  the  MERIT  output  is  in  Appendix  E. 
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D.3.5  Listing 

The  MERIT  source  listing  is  presented  in  Table  D-IX. 

D.4  RANK 
D.4.1  Purpose 

The  RANK  program  was  required  to  analyze  a large  number  of 
individual  rankings  obtained  from  the  MPP  Impact  Evaluation  survey  and 
produce  a composite  ranking. 

D.4. 2 Input 

RANK  accepts  as  input  N individual  rank  orderings  in  the  form  of 
vectors  containing  the  integers  1,2, M in  any  order. 

D.4. 3 Function 

For  each  of  the  M positions  (i.e.,  the  length  of  the  input  rank 
vectors),  RANK  computes  the  mean,  the  variance  and  upper  and  lower  bounds 
for  the  rank  deviation.  RANK  also  computes  other  statistics,  including 
the  Kendall  Coefficient  of  Concordance  and  the  Spearman  Rank  Correlation 
Coefficient  for  each  pair  of  ranking  vectors,  and  sorts  on  the  mean  rank 
values  to  obtain  the  composite  rank  ordering. 

D.4. 4 Output 

RANK  output  consists  of  the  composite  rank  ordering  vector,  mean 
rank  values,  variances  and  upper  and  lower  bounds  of  rank  deviation  and 
the  correlation  coefficients.  Two  examples  of  the  mean  rank  and  composite 
rank  ordering  results  for  ranked  software  development  problems  and  ranked 
MPP  are  contained  in  Tables  D-X  and  D-XI. 

D.4. 5 Listing 

The  RANK  source  listing  is  presented  in  Table  D-XI I . 
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APPENDIX  E 


DETAILED  MPP  COMPARISON  RESULTS 


E. 1 Contents 

This  appendix  contains  output  of  the  MERIT  support  program.  MERIT 
was  used  to  determine  the  effect  of  applying  a selected  set  of  weights  to 
the  raw  response-frequency-distribution  data  from  the  MPP  impact  evaluation 
survey.  This  was  done  (as  discussed  in  Section  4 of  this  report)  to  support 
comparative  evaluation  of  two  MPP  implementations.  The  MPP  comparison 
methodology  was  demonstrated  via  determination  of  figures  of  merit  for  two 
disjoint  subsets  of  the  Systems  Technology  Program  MPP. 

E.2  Format 

The  format  of  the  MERIT  output  presented  on  Tables  E-I  and  E-II  is 
described  in  detail  in  Section  D.3.4  (Appendix  D).  Table  E-I  presents  MERIT 
output  for  MPP  Set  #1  consisting  of  the  practices  numbered  1 through  5, 
while  E-II  is  for  MPP  Set  #2  consisting  of  the  practices  numbered  6 through 
11. 
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METRIC  SYSTEM 


BASE  UNITS: 

Quantity 


length 

mass 

time 

electric  current 
thermodynamic  temperature 
amount  of  substance 
luminous  intensity 

SUPPLEMENTARY  UNITS: 

plane  angle 
solid  angle 

DERIVED  UNITS: 
Acceleration 

activity  (of  a radioactive  source) 

angular  acceleration 

angular  velocity 

area 

density 

electric  capacitance 
electrical  conductance 
electric  field  strength 
electric  inductance 
electric  potential  difference 
electric  resistance 
electromotive  force 
energy 
entropy 
force 

frequency 
illuminance 
luminance 
luminous  flux 
magnetic  field  strength 
magnetic  flux 
magnetic  flux  density 
magnetomotive  force 
power 
pressure 

quantity  of  electricity 
quantity  of  heat 
radiant  intensity 
specific  heat 
stress 

thermal  conductivity 
velocity 

viscosity,  dynamic 

viscosity,  kinematic 

voltage 

volume 

wavenumber 

work 


SI  PREFIXES: 


Unit 


metre 

kilogram 

second 

ampere 

kelvin 

mole 

i.andela 


radian 

steradian 


metre  per  second  squared 

disintegration  per  second 

radian  per  second  squared 

radian  per  second 

square  metre 

kilogram  per  cubic  metre 

farad 

siemens 

volt  per  metre 

henry 

volt 

ohm 

volt 

joule 

joule  per  kelvin 

newton 

hertz 

lux 

candela  per  square  metre 
lumen 

ampere  per  metre 

weber 

tesla 

ampere 

watt 

pascal 

coulomb 

joule 

watt  per  steradian 

joule  per  kilogram-kelvin 

pascal 

watt  per  metre-kelvin 
metre  per  second 
pascal-second 
square  metre  per  second 
volt 

cubic  metre 
reciprocal  metre 
joule 


SI  Symbol 

Formula 

m 

kg 

s 

A 

K 

mol 

cd 

rad 

sr 

m/s 

(disintegration  )/s 
rad/s 

rad/s 

m 

kg/m 

F 

A-s/V 

S 

MV 

VI  m 

II 

V-s/A 

V 

W'A 

VIA 

V 

W/A 

1 

N-m 

J/K 

N 

kg-m/a 

Hz 

(cycle)/a 

lx 

Im/m 

cd/m 

Im 

cd-ar 

A/m 

Wb 

V-a 

T 

Wb/m 

A 

W 

)/» 

Pa 

N/m 

C 

1 

A-s 

N-m 

W/sr 

J/kg-K 

Pa 

N/m 

VV/m-K 

m/s 

Pa-s 

m's 

V 

W/A 

m 

1 

(wave)/m 

N-m 

Multiplication  Factors 

Prefix 

1 000  00 0 000  000 

= to1' 

tnra 

1 O00  000  000 

= HI’ 

gig* 

1 000  000 

= HI" 

mega 

1 000 

= to’ 

kilo 

100 

= w1 

hecto* 

10 

» 10’ 

(Inks' 

0 1 

= 10-’ 

duel* 

0 01 

= 10-J 

centi* 

0 001 

= 10"’ 

mill! 

0 000  001 

= 10-* 

micro 

0 000  000  001 

* 10-* 

nann 

0 000  000  000  001 

= 10- ,a 

pico 

0 000  000  000  000  001 

* 10-” 

fomto 

o ooo  ooo  ooo  ooo  ooo  noi 

= 10-’* 

atto 

* To  be  avoided  where  possible 


SI  Symbol 


T 

(2 

M 

k 

h 

da 

d 


i: 


n 


? 


